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FOREWORD 


The  effective  performance  of  Navy  systems  in  Fleet  use  is 
essential  to  successful  Naval  operations.  With  the  increasing 
complexity  of  Naval  systems,  the  achievement  of  an  acceptable  level 
of  performance  effectiveness  has  become  increasingly  difficult. 
During  the  past  few  years,  useful  techniques  have  evolved  from  a 
continuing  extensive  effort  to  develop  the  ways  and  means  to  inprove 
performance  effectiveness  of  Fleet  systems. 

The  purpose  of  this  manual  is  to  describe  the  techniques 
developed  under  the  Systems  Performance  Effectiveness  (SFE) 
program.  Navy  program  managers,  project  managers  and  project 
engineers  are  urged  to  use  this  manual  as  appropriate  in  the 
development  of  systems  to  assure  the  maximum  feasible  effectiveness. 
Distribution  of  this  manual  to  industrial  organizations  as  necessaiy 
and  appropriate  is  authorized. 

This  revised  manual  is  the  first  updated  edition  of  NAVMAT 
?394l  dated  May  1967  (AD660  4l3)» changes  from  the  basic  manual 
are  indicated  by  the  following  marginal  symbols: 

Line  changed 

Lines  changed 

It  i3  intended  that  the  publication  will  continue  to  be 
periodically  revised  and  updated  to  meet  the  varied  needs  of  groups 
within  the  Naval  Material  Command.  Comments  or  recommendations 
concerning  the  content  of  this  manual  should  be  forwarded  to  the 
Chief  of  Naval  Material  (MAT  0325)  at  any  time  for  consideration. 
This  publication  has  been  reviewed  and  approved  in  compliance  with 
SECNAVINST  5600.16. 


^ULdru) 

Deputy  Chief  of  Material  for 

Development/ Chief  of  Naval  Development 
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AN  INTRODUCTION  TO 
SYSTEMS  PERFORMANCE  EFFECTIVENESS 

0  N  E 


1.1  BACKGROUND 

The  effective  performance  of  systems  in  the  Fleet  has  always  been  an 
important  factor  in  the  successful  accomplishment  of  the  Navy's  missions.  In 
recent  years,  however,  the  tremendous  growth  in  system  complexity,  due  to  more 
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FIGURE  1-1 

COMPLEXITY  INCREASE,  AVIONICS  SYSTEMS 


complex  and  demanding  operational 
requirements  and  to  rapidly  advancing 
technology,  has  magnified  the  problem 
of  obtaining  effective  performance 
from  the  new  systems  being  delivered. 
For  example,  Figure  1-1  depicts  the 
rapid  growth  of  aircraft  electronics, 
measured  in  terms  of  active  element 
groups  (diodes,  transistors,  and  vacuum 
tubes,  together  with  their  associated 
passive  components).  In  the  12  years 
from  1953  to  1965  the  complexity  of 
avionic  systems  increased  by  a  factor 
of  7k,  although  improvements  in 
technology  and  techniques  permitted 
the  attainment  of  this  complexity  with 
a  weight -increase  factor  of  only  3*8. 

A  large  avionic  system  now  planned 
for  1968  will  be  160  times  more  complex 
than  a  1953  avionic  system. 
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FIGURE  1-2 

COMPLEXITY  INCREASE,  DESTROYER 
ELECTRONICS  SUITS 
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A  similar  increase  in  complexity 
is  evident  in  shipboard  electronics, 
as  illustrated  in  Figure  1-2.  The 
figure  shows  the  increase  in  system 
complexity — measured  by  the  number  of 
active  element  groups — in  destroyer 
"electronics  suits"  over  the  period 
1937  to  1964.  Figure  1-3  relates  this 
increase  over  the  last  twenty  years 
to  the  corresponding  increases  in  cost, 
power,  weight,  and  volume.  It  is 
interesting  to  note  that  although 
complexity  has  increased  by  a  factor 
of  40,  technological  progress  has 
resulted  in  smaller  increases  in  the 
other  factors:  a  factor  of  21  for 
cost,  20  for  power,  14  for  weight, 
and  12  for  volume. 
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FIGURE  1-3 

INCREASES  IN  CHARACTERISTICS 
OF  DESTROYER  ELECTRONIC  SUITS 


The  complexity  explosion  has  not 
been  restricted  to  electronic  systems, 
however.  Threats  are  more  complex, 
decision  time  is  reduced,  tactical 
decisions  are  more  critical,  available 
reaction  time  Is  reduced;  all  these 
'squeeze1  factors  tend  to  require 
increased  complexity  in  all  the  systems 
that  combine  to  form  a  capability. 

The  seemingly  simple  task  of  collecting 
feedback  data,  for  example,  has  escalated 
to  a  point  where  the  complex  Naval 
Tactical  Data  System  is  required. 

More  complex  systems  in  turn  generate 

greater  technical  and  operational  interface  problems.  Consequently,  the  achieve¬ 
ment  of  satisfactory  systems  performance  effectiveness  in  the  Fleet  has  challenged 
the  ingenuity  of  the  scientist,  the  engineer,  and  the  technical  manager,  and  has 
influenced  trends  in  all  the  engineering  and  management  disciplines. 

Although  special  attention  during  the  past  several  years  to  such  areas 
critical  to  systems  performance  effectiveness  as  reliability,  maintainability, 
performance,  operability,  quality,  human  factors,  safety,  and  value  has  provided 
some  measure  of  Improvement  in  each  area,  further  improvements  in  these  techniques 
and  the  development  of  methods  for  efficiently  integrating  them  into  an  effective¬ 
ness  discipline  are  essential. 


0  \ 


1.2  DEFINITION  AND  AIMS  OF  SYSTEMS  PERFORMANCE  EFFECTIVENESS 

Systems  performance  effectiveness  can  be  defined  as,  "A  measure  of  the  extent 
to  which  a  system  can  be  expected  to  complete  its  assigned  mission  within  an 
established  time  frame  under  stated  environmental  conditions"*.  This  definition 
has  received  general  acceptance,  and  accurately  establishes  the  scope  and  context 
of  the  discipline.  Although  the  word  "measure"  is  ambiguous --here  assumed  to 
imply  a  standard  of  comparison  or  judgement — the  other  terms  clearly  indicate 
that  effects  of  mission  environment  and  the  time  demands  of  system  operation 
are  an  integral  part  of  the  systems  performance  effectiveness  concept. 

A  meaningful  concept,  however,  requires  more  than  acceptable  definitions. 

To  provide  an  analysis  capability,  a  structure  or  frame  of  reference  is  required 
for  the  principle  elements  of  the  concept;  the  nature  of  the  interrelationships 
among  the  elements  must  be  Identified;  and  a  set  of  techniques  must  be  developed. 
Rules  and  procedures  for  applying  the  techniques  are  desirable. 

The  purpose  of  this  manual  is  to  present  the  concept  and  highlights  of 
systems  performance  effectiveness  and  the  steps  required  for  implementation. 

In  general,  there  is  a  growing  volume  of  literature  on  systems  effectiveness. 

Some  Important  reference  sources  are  cited  at  appropriate  points  in  the  manual. 

Appendix  A  contains  a  description  of  the  nature  of  the  acquisition  process — 
the  process  for  which  systems  performance  effectiveness  serves  as  a  measure  and 
a  tool.  The  role  and  utility  of  systems  performance  effectiveness  modeling 
techniques  during  various  phases  of  the  acquisition  process  is  also  discussed  in 
this  appendix.  Considerations  relating  to  characteristics  and  attributes  of 
analytic  models,  and  discussions  on  the  formulation  of  an  appropriate  analysis 
framework  for  systems  performance  effectiveness  are  presented  in  Appendix  E. 


•  System*  Effectiveness,  cospiled  by  Systems  Effectiveness  Branch, 
Office  of  Naval  Material,  January  1965  (AD  659-520). 
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IMPLEMENTATION  OF 
SYSTEMS  PERFORMANCE  EFFECTIVENESS 


T  W  0 


Several  technical  disciplines  are  involved  in  implementing  a  disciplined 
approach  to  achieving  satisfactory  systems  performance  effectiveness.  In  general 
they  include  technical  documentation,  data  acquisition  and  reduction,  system 
evaluation,  analysis,  and  systems  engineering. 

The  natural  tendency  to  seek  a  unifom,  optimum  criteria  for  system 
development  actually  may  not  be  the  best  approach.*  The  systems  performance 
effectiveness  model  can  take  several  forms;  depending  on  the  mission  requirements, 
complexity,  and  development  status  of  the  system  or  equipment  in  question.  The 
problem,  then,  can  be  stated  as  follows:  How  is  effective  systems  engineering 
performed  in  a  real-world  environment?  The  techniques  of  systems  performance 
effectiveness  have  evolved  from  the  endeavors  to  answer  thi3  question. 

2  1  SYSTEMS  PERFORMANCE  EFFECTIVENESS  FACTORS 

The  use  of  an  effectiveness  framework  permits  a  multiplicity  of  missions, 
functional  considerations,  and  resource  considerations  to  be  analyzed  within  a 
common  frame  of  reference.  The  elements  from  which  the  Navy's  PAU  effectiveness 
framework  13  synthesized  is  illustrated  in  Figure  2-1;  the  framework  is  described 
In  detail  In  Appendix  B.  Figure  2-1  indicates  the  structural  relationships 

*"We  cannot  and  should  not  expect  ever  to  develop  a  complete  set  ct  numerical 
criteria  to  measure  military  effectiveness",  from  a  McNamara-Hltch-Fr.thoven 
Anthology,  A  Modem  Design  for  Defense  Decision,  p.  l?k.  Industrial  d.  llege 
of  the  Armed  Forces,  July  l^hb. 


among  the  PAU  elements  and  the  number  of  factors  each  contains.  The  Inclusion 
of  elements  relating  to  military  worth  and  system  penalties  allows  the  structure 
to  be  expanded  to  higher  levels.  For  the  purposes  of  a  systems  performance 
effectiveness  analysis.  Figure  2-1  basically  serves  as  a  checklist  for  iden¬ 
tifying  the  particular  set  of  factors  of  concern  for  a  given  problem;  the 
significance  of  each  factor  will  vary  from  problem  to  problem.  The  framework 
provides  initial  perspective  and  organizes  information  to  be  used  in  subsequent 
analysis. 

If  the  system  performance  effectiveness  discipline  is  to  represent  a  useful 
solution  to  the  project  manager's  problems,  performance  parameters  that  are  speci¬ 
fied  in  procurement  contracts  and  the  associated  effectiveness  factors  must  be 
related  to  system  needs.  Thus,  the  planning  sequence  must  lead  to  a  definl  ti  ... 
of  critical  performance  parameters  such  that  the  requirements  developed  during 
Concept  Formulation  and  Contract  Definition  are  compatible  with  the  technical 
system  design  and  can  be  analyzed  by  a  systems  performance  effectiveness  model. 

2.2  DEVELOPMENT  OF  EFFECTIVENESS  CRITERIA 

Conversion  of  the  systems  performance  effectiveness  structure  into  a  practical 
technique  is  discussed  in  this  section.  Considerations  and  procedures  leading 
to  a  formulation  of  effectiveness  criteria  and  characteristics  and  appropriate 
levels  of  abstractions  to  be  used  in  generating  the  criteria  are  discussed. 

Figure  2-2  outlines  the  complexity  of  developing  systems  performance  effec¬ 
tiveness  criteria  for  a  particular  system.  In  threat  analysis  the  Droject  manager 
must  consider  that  both  weapon-system  technology  and  threat  capability  are  in  a 
dynamic  state.  This  dynamic  situation  is  a  result  of  an  evolving  state  of  the 
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art}  and  the  fact  that  the  problem  often  requires  analysis  of 
sequential  series  of  events  (mission  scenarios)  whose  factors  are 
not  easily  related  analytically.  Simulation  is  therefore  a  useful 
tool  for  modeling  this  problem.  Inputs  associated  with  the  user 
play  a  significant  role  in  the  analysis,  and  special  agencies  such 
as  the  Center  for  Navel  Analyses  are  employed  to  provide  operations- 
research  (OR)  analyses.  In  mission  analysis  the  manager  must  make 
subjective  decisions  to  reduce  the  problem  to  manageable  levels 
without  causing  loss  of  significance  in  the  results.  The  tasks 
he  selects  for  inclusion  in  the  model  must  be  those  whose  successful 
performance  accurately  represents  mission  success.  Furthermore, 
to  assess  total  costs,  personnel  and  other  data,  the  complete  set 
of  functions  and  equipments  to  accomplish  the  tasks  should  be 
considered. 

It  is  important  that  the  user,  producer,  and  review  agency  all 
agree  on  the  functional  requirements  specified  in  the  task  descriptions. 
These  task  descriptions  form  the  basis  of  effectiveness  analysis. 
Effectiveness  measures  that  relate  successful  performance  of  the  task 
sure  generated:  the  measures  form  the  basis  for  analytic  modeling  to 
assess  sensitivities  of  selected  parameters  to  resources,  failures, 
logistics,  degradation,  time  demands,  etc.  The  effectiveness 
measures  can  generally  be  divided  into  two  distinct  but  related 
factors.  The  first  factor  relating  to  the  performance  variable  and 
its  associated  range  of  values,  and  the  second  describing  the  time 
dependency  of  the  performance  values.  The  bottom  line  of  Figure  2-2 
expresses  analytically  the  performance  capability  (Pq)  and  the 
detailed  time-dependency  of  performance  (Pp).  adds  a  time 
dimension  to  performance  and  allows  reliability,  maintainability, 
availability,  etc.,  to  be  treated  as  modifiers  of  performance. 
Effectiveness  then  can  be  analytically  measured  as  a  function  of 
Pc  and  PT.  The  effectiveness  criteria  ultimately  selected  must  meet 
the  following  requirements: 

•  They  must  be  quantitative. 

•  They  must  be  sufficiently  meaningful  to  the  system  designer 
and  system  analyst  to  permit  their  use  as  design  and 
evaluation  criteria  for  a  given  system. 

•  They  must  be  sufficiently  meaningful  to  the  user  and  mission 
analyst  to  permit  their  value  to  be  specified  and  interpreted 
in  terms  of  the  mission. 

An  important  stage  in  the  development  of  effectiveness  criteria  is 
the  estimating  of  the  resource  constraints  for  each  system.  These  are 
treated  in  three  distinct  but  logical  ways  by  the  project  manager. 

First,  he  places  absolute  upper  limits  on  the  particular  constraints 
that  are  specified.  For  example,  Program  A  may  have  a  two-year  lead 
time  as  sn  absolute  requirement;  Program  B  may  have  certain  size  and 

weight  constraints  because  of  payload  restrictions,  etc.  These  limits 
are  treated  in  a  go/no-go  sense:  only  systems  which  satisfy  these 
requirements  can  be  candidates. 
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Second,  the  constraints  are  treated  as  design  parameters,  and  the  impact 
of  each  is  measured  where  possible  as  a  change  in  f(Pc,  PT).  For  example,  employ¬ 
ment  of  technicians  in  the  higher  skill  brackets  reduces  downtime  and  therefore 
affects  Ppj  provision  of  ample  lead  time  reduces  development  risk,  allows  advanced 
technology  to  be  employed,  and  therefore  affects  Pc. 

Finally,  the  constraints  are  treated  as  quantitative  attributes  of  the  system 
in  their  own  right,  and  specific  values  are  estimated  in  terms  of  years  and  dollars 
for  each  design  approach.  Figure  2-3  gives  an  overview  of  the  use  of  (Pc,  PT) 
effectiveness  criteria  used  in  developing  resources  as  a  function  of  the  Mainten¬ 
ance  Engineering  Analysis  in  the  Integrated  Logistic  Support  Planning  procedure. 


FIGURE  2-3 

EFFECTIVENESS  CRITERIA  |P  P  )  FOR  IIS* 


♦NAVKAT  INST  4000.20,  Integrated  Logistic  Suppo 
Procedures,  19  August  i960. 


rt  Pli 


The  concept  of  (Pq,  Pp)  is  not  a  figure  of  merit  approach  but  rather 
allow"'  effectiveness  analysis  to  be  treated  in  a  sequential  manner  and  provides 
visibility  of  pertinent  information  to  enable  decision  making. 

The  attempt  to  combine  effectiveness  measures  into  a  single  figure  of  merit 
poses  a  question  currently  receiving  much  attention:  To  what  level  should  the 
elements  be  combined?  For  example,  if  a  relative  value  can  be  placed  on  cost 
versus  manpower  skills,  size,  etc.,  it  is  possible  to  combine  the  elements  into 
a  single  cost  figure  of  merit.  In  general,  however,  a  better  approach  is  not 
to  combine  data  beyond  the  point  at  which  useful  information  is  lost.  For 
instance,  the  ability  to  study  systems  from  the  viewpoint  of  individually  esti¬ 
mated  resource  constraints  permits  analysis  to  reflect  change  in  emphasis  at  any 
point  in  time.  Figure  2-4  is  a  symbolic  presentation  of  system  effectiveness  for 

a  large  system:  effectiveness  appears 
in  a  series  of  relationships  that  show 
the  lead  time  (1^,),  manpower  (Mp), 
acquisition  cost  (CA),  ownership  cost 
(CQ),  etc.,  as  modifiers  of  an  analytic 
representation  of  effectiveness.  The 
factor  W,  a  measure  of  mission  worth, 
is  Introduced  to  illustrate  that  even 
this  general  form  is  subject  to  addi¬ 
tional  variation  depending  on  whether 
the  tasks  of  multitask  missions  are 
treated  independently  or  In  a  combined 
form. 

2.3  TECHNICAL  COMMUNICATIONS  AND 
ANALYSIS 

An  analogy  can  be  drawn  between 
the  PAU  framework  and  associated 
factors,  and  the  Pc,  PT  expression  and  its  associated  techniques.  The  techniques 
associated  with  the  Pfi,  PT  expression,  however  encompass  not  only  elements  of 
analysis — such  as  prediction,  assessment,  trade-off,  and  optimization  routines — 
but  deal  with  methods  that  have  been  developed  to  help  achieve  adequate  technical 
communications  among  all  system-acquisition  participants. 

The  tools  needed  for  meaningful  Implementation  of  systems  performance 
effectiveness,  then,  may  be  divided  into  two  broad  categories.  Those  in  the  first 
category  provide  the  means  of  achieving  adequate  technical  communications  among 
all  participants  in  a  system's  acquisition;  chose  in  the  second  category  provide 
the  means  for  performing  systems  performance  effectiveness  analysis.  These 
elements  have  been  the  objects  of  comprehensive  studies  leading  to  the  development  of 
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FIGURE  2-4 

ANALYTIC  REPRESENTATION  OF  NAVY  MODEL 


techniques  which,  when  used  In  combination,  provide  some  of  the  basic  ingredients 
of  the  practical  systems  performance  effectiveness  methodology  known  as  the  General¬ 
ized  Effectiveness  Methodology  (GEM).  GEM,  shown  in  schematic  form  in  Figure  2-5, 
is  basically  an  analytic  method  of  addressing  the  systems  performance  effective¬ 
ness  problem.#  The  organization,  logic,  current  status  of  development,  and 
capabilities  of  GEM  are  described  in  Section  2.3.2,  While  GEM  has  been  specifi¬ 
cally  designed  for  easy  access  and  use,  caution  must  be  exercised  in  the  use  of 
its  more  specialized  features.  In  some  cases  it  may  be  necessary  to  verify  that 
the  problem  to  be  solved  and  a  particular  GEM  technique  are  compatible.  It  is 
recommended  that  only  experienced  personnel  make  this  determination.  # 

The  two  broad  aspects  of  systems  performance  effectiveness — Technical 
Communications  and  Analysis — are  discussed  in  Sections  2.3.1  and  2.3.2  respectively. 


FIGURE  2-5 

GENERALIZED  EFFECTIVENESS  METHODOLOGY 


*  U.S.  Naval  Applied  Science  Laboratory,  System  Effectiveness  Branch, 
Brooklyn,  New  York  11251.  Major  components  of  (SIM  have  been  implemented 
on  a  CDC  6600  computer. 

**  The  Generalized  Effectiveness  Methodology  (GEM)  Analysis  Program,  NASL 
Progress  Report  1,  Lab.  Project  920-72-1,  8  May  1968  (AD  832-107L) 
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2.3.1  Technical  Communications 


The  Need 

Technical  communications  probably  form  the  most  Important  requirement  for 
a  successful  system-development  program:  criteria  development  emerges  as 
primarily  a  dialogue  between  user  and  producer  (see  Appendix  A).  The  establishment 
of  formal  concept-formulation  and  contract-definition  outputs,  whose  content  is 
specified  in  documents  such  as  DOD  Directive  3200.9,  are  based  upon  this  need. 

The  diversity  in  the  size  and  complexity  of  equipments  purchased  by  the 
Navy  reautres  a  corresponding  diversity  in  acquisition-management  structures. 
j.n  turn,  technical  communication  must  be  geared  to  accommodate  these  diversities. 
Information-disclosure  formats,  therefore,  must  have  sufficient  commonality  to 
permit  structuring  of  various  subsystem  organizations  for  various  management  levels. 
Furthermore,  since  system  acquisition  is  time  dependent,  technical  communications 
must  be  geared  to  the  development  cycle  and  grow  with  the  system.  If  communi¬ 
cation  is  not  adequate,  high  system  complexity  and  the  dep\  t  of  the  organization 
required  to  manage  the  program  will  result  in  allocation  of  goals  and  sub- 
optimization  of  goals  without  the  benefit  of  the  total  systems  approach.  The 
communications  requirement,  therefore,  must  be  specifically  defined  in  terms  of 
what  Is  to  be  communicated  and  how  it  is  to  be  communicated. 

The  requirement  is  illustrated  in  Figure  2-6  and  may  be  summarized  as  follows: 


i! 
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.  The  ability  to  handle  change  (update  capability) 

«  The  ability  to  sort  information  (reference) 

•  The  ability  to  Integrate  information  at  different  levels  and  from 
different  sources  (commonality) 

•  The  ability  to  provide  critical  inputs  to  the  modeling  effort  (data) 
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A  Method:  Design  Disclosure* 


A  technical-communications  scheme  called  the  Design  Disclosure  for 
Systems  and  Equipment  (DDSE)  has  been  developed  in  response  to  the  described 
need.  It  is  based  on  NASL  Technical  Memorandum  5  of  15  March  1967  (Lab.  Pro¬ 
ject  920-72-2),  and  is  contained  in  Military  Standardization  Handbook 
MIL-HDBK-226  (NAVY)  sponsored  by  the  Naval  Ship  Systems  Command.**  It  provides 
for  the  transfer  of  design  and  development  information  from  the  function  and 
hardware  designers  to  the  system  designers,  systems  analysts,  mission  analysts, 
and  users.  For  this  purpose  it  specifies  the  use  of  four  information- transfer 
structures:  block  diagrams,  blocked  schematics,  blocked  texts,  and  design 
outlines. 


The  detailed  block  diagrams  and  the  blocked  schematics  define  the  boundaries 
— ►-  of  functional  and  hardware  entities.  The  blocked- teftt  structure  is  an  economical 
way  of  increasing  management  control  and  understanding;  it  presents  theory, 

— ►  performance  data,  goals,  etc.,  on  a  page  facing  the  block  diagrams  and  blocked 
schematics.  The  information— in  detail  applicable  to  the  particular  level— is 
inserted  appropriately  in  blocks  which  comprise  a  mirror  image  of  the  detailed 
block  diagram  and  blocked  schematics.  Lastly,  the  design  outline  presents  all 
the  time  and  information  dependencies  at  any  level  for  all  the  hardware  and 
functional  entities  in  all  operating  modes. 

— ►-  Figure  2-7  is  an  overall  summary  of  the  acquisition  adaptation  of  the  DDSE. 

2.3.2  Analysis:  The  GEM  Program 

In  this  discussion,  analysis  is  interpreted  as  a  procedure  for  "providing 
the  best  possible  estimate  of  the  effects  of  selecting  various  courses  of 
— ►  action"***.  While  there  are  many  reasonable  courses  of  action  within  the  frame- 
— ^  work  of  system  performance  effectiveness,  the  approach  selected  attempts  to 

incorporate  as  much  problem  solving  capability  as  is  consistent  with  the  current 
state  of  the  art.  Furthermore,  it  maintains  a  sufficiently  flexible  structure 
and  logic  in  the  model  to  allow  for  further  expansion  and  introduction  of 
additional  capability  when  new  techniques  are  developed  and  their  impact  on 
systems  performance  effectiveness  has  been  ascertained.  Figure  2-5  showed  the 
logic  and  main  components  of  the  approach  selected:  the  Generalized  Effectiveness 
Methodology  (GEM).  The  various  elements  depicted  in  the  figure  are  discussed 
below. 

User  Directions 

The  user  directions  (l)  provide  the  means  for  describing  the  system 
(Definition  Language)  and  (2)  direct  modifications  and  evaluation  of  the  model 
(Command  Language). 

Definition  Language  -  The  definition  language  is  used  to  describe 
‘-.V.*  system  (network  or  block  diagram).  The  system  may  contain  combinations  of 
series  parallel  configurations;  shared  elements  or  bridge  networks  may  be 

nr  ♦Originally  termed  Design  Disclosure  Format (DDF)  and  Design  Disclosure  Standard(DDS) 

**  Analogous  and  compatible  technical  c  inn  1  cation  techniques  used  in  Symbolic 
Integrated  Maintenance  Manuals  (SUM)  are  described  in  Mil.  Spec.  MIL-M-2410QA 
(SHIPS). 


***  Hitch,  Decision  Maki 
Los  Angeles,  19^5^ 


for  Defense,  University  of  California  Press, 


described  by  Boolean  (logical)  functions.  The  processor  program*,  in 
association  with  the  description,  generates  the  appropriate  formulas  or 
differential  equations  and  a  computer  program  to  evaluate  the  model. 

Command  Language  -  The  command  language  Is  used  to  direct  evaluation 
of  a  system  in  terms  of  any  or  all  the  variables  that  have  been  Indicated  in  the 
system  definition.  Also  provided  in  this  language  are  commands  which  are  used 
to  modify  the  system  definition,  evaluate  the  resulting  definition,  and  thereby 
generate  alternatives  to  the  original  system. 

By  means  of  the  commands- -ADD,  DELETE,  REPLACE,  ALTER,  VARY- -the  user  is  able 
to  add,  delete,  or  replace  items  in  the  system  configuration;  alter  any  formula 
reference;  and  vary  any  single  argument  value  or  set  of  argument  values  throughout 
the  system.  The  command  "VARY"  permits  the  user  to  generate  automatically  the 
information  needed  for  a  sensitivity  analysis. 

Technical  Communications  for  Systems  Under  Development 

The  system  that  is  being  analyzed  must  be  described  in  terms  appropriate 
to  the  analysis  to  be  performed  (e.g.,  reliability  analysis  requires  a  reliability 
network  of  the  system).  Currently  this  represents  a  complex  and  tedious  procedure 
involving  the  services  of  specialists  from  diverse  disciplines.  Use  of  a  DDS 
description  of  the  system  and  methods  for  extracting  the  information  pertinent  to 
the  particular  analysis  under  study,  will  simplify  and  expedite  the  task.  The 
information  extracted  can  then  be  coded,  either  manually  or  semi-automatieally, 
into  OEM  language  and  made  to  represent  the  required  input  to  the  program. 

Library  of  Equations 

The  library  of  equations  (formula  library)  consists  of  a  set  of  computer 
programs  for  evaluating  the  functions  most  frequently  used  when  the  variables 
pertinent  to  systems  performance  effectiveness  are  described. 

The  equations  that  are  now  available  will  enable  the  program  manager  to 
compute  the  following  variable*: 

(1)  Reliability  Without  Repair 

•  In  general,  the  reliabillty-withovt-repalr  variable  can 

be  calculated  for  exponential,  Weibull,  gamma,  log-normal,  and 
truncated  normal  failure  distributions. 

•  For  parallel-standby  configurations,  the  reliability-without-repair 
variable  can  be  computed  for  exponential  and  Weibull  failure 
distributions. 

•  The  rellabillty-wlthout-repalr  variable  can  be  computed  for  a  system 
that  comprises  several  identical  elements  arranged  linearly  or  In  a 
circular  pattern  (such  as  in  a  cylindrical  sonar  transducer). 

TUee",(flK  Processor,  page  18 
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complete  disclosure  and  provides  a  useful  vehicle  lor 
folloaHin  production  runs.  The  ccntent  of  the  format 
pxkage  can  no*  ht  used  as  a  basis  lor  procurements  ol 
similar  system-equipments. 

In  addition,  this  final  package  together  with  the  previous 
revie*  packages  provides  a  detailed  history  of  th« 
•volution  ot  the  system-equipment. 


FIGURE  2-7  DESIGN  DISCLOSURE  LIFE  CYCLE  ADAPTATIO 
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(2)  Reliability  With  Repair 

The  reliability-with-repair  variable  can  be  computed  for  the  following 
combinations  of  failure  and  repair  distributions: 

•  Exponential  failure  -  exponential  repair 

•  Weibull  failure  -  exponential  repair 

•  Weibull  failure  -  lognormal  repair 

Under  the  exponential-exponential  condition,  the  program  will  also 
account  for  the  following  circumstances: 

•  Limited  number  of  repairmen  and  a  repair  priority 

•  Limited  number  of  spares  assigned  to  specific  equipments  or  pooled 
among  equipments 

•  Combinations  of  the  above, 

In  the  cases  of  Weibull  failure  and  Exponential  or  Lognormal  repair, 
there  are  several  equipment-configuration  and  maintenance-personnel 
restrictions  imposed. 

(3)  Availability 

The  availability  variable  can  be  computed  for  the  following 

combinations  of  failure  and  repair  distributions:  [see  restrictions  under 

(2)] 

•  Exponential  failure  -  exponential  repair 

•  Weibull  failure  -  exponential  repair 

•  Weibull  failure  -  lognormal  repair 

(4)  Interval  Reliability 

Interval  Reliability  is  a  major  component  of  the  measure  of  effective¬ 
ness.  It  is  defined  as  the  probability  that  the  system  is  in  satisfactory 
condition  at  seme  time,  t^,  (mission  start)  and  does  not  fail  during  a 
time  Interval,  t,  following  t-^.  The  interval  reliability  of  a  system 
can  be  calculated  for  conditions  identical  to  those  described  under 
Reliability  with  Repair  and  Availability. 

(5)  Steady  State  Availability 

The  steady  state  availability  of  a  system  can  be  calculated  for  the 
combination  of  exponential  failure  and  exponential  repair  only, 

(6)  Mission  Availability 

The  mission  availability  of  a  system  can  be  calculated  for  the 
combination  of  exponential  failure  and  exponential  repair  only. 

(7)  Mean  Life 

The  aean-life-vithout- repair  parameter  can  be  calculated  for  any 
combination  of  failure  distributions.  The  aean-life-vith- repair 
parameter  can  be  calculated  only  for  the  combination  of 
exponential  failure  and  exponential  repair. 
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An  update  program  for  the  formula  library  is  also  provided  so  that 
the  user  can  incorporate  new  formulas  into  the  library  and  delete  ones  no 
longer  of  interest.  In  addition  to  expanding  the  set  of  formulas  related 
to  systems  performance  effectiveness  variables ,  new  variables  may  be 
named  and  formulas  related  to  these  variables  may  be  added.  Thus,  when 
quantitative  relationships  for  variables  such  as  maintainability  are 
developed,  they  can  be  added  to  the  formula  library. 

GEM  Processor 


The  GEM  processor  is  a  computer  program  which  generates  the 
mathematical  routines  for  the  systems  and  its  alternatives,  and  executes 
the  coded  program.  If  any  item  has  been  described  by  a  Boolean  formula, 
the  processor  will  generate  the  appropriate  probability  formula  for  use 
in  the  model. 

Stored  Networks 


Any  system  description  which  has  been  processed  by  GEM  can  be  stored 
and  subsequently  retrieved.  The  stored- networks  library  provides  a  fast- 
access  record  of  all  analyzed  systems. 

An  update  program  for  the  stored-networks  library  is  also  provided 
to  generate,  update,  and  maintain  a  System  Library  free  from  errc^.  The 
update  program  provides  the  user  with  the  following  capabilities: 

•  Rename  a  present  system  in  the  library  (RENAME) 

•  Delete  a  present  system  from  the  library  (ERASE) 

•  Add  a  new  system  to  the  library  (COPT) 

•  Delete  variable (s)  from  a  present  system  in  the  library  (EXCLUDE) 

•  Add  a  new  variable(s)  to  a  system  in  the  library  (INCLUDE) 

Common  Query  System 

Having  established  the  system  description  and  defined  the  appropriate 
measures,  it  is  necessary  to  introduce  data  to  obtain  quantitative 
estimates.  In  the  past  decade,  the  availability  of  data  has  improved 
considerably.  Methods  of  storing  and  retrieving  information  have 
advanced,  and  determined  efforts  have  been  made  to  collect  accurate  design 
and  operational  data.  In  addition  to  collecting  performance,  reliability, 
and  maintainability  information,  data  banks  are  now  recording  information 
on  items,  such  as  costs  ar.**  personnel.  While  an  extensive  number  of 
data  sources  have  been  est«olished,  there  is  little  similarity  exhibited 
with  regard  to  either  collection  procedures,  storage  medium  used,  or 
methods  available  for  retrieval.  This  situation  exists  because  the 
individual  activities  perform  different  functions  and  tend  to  reflect  the 
emphasis  and  interest  each  agency  places  on  its  data.  In  instances  where 
data  requirements  can  adequately  and  consistently  be  met  by  a  single  source, 
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the  existence  of  other  data  centers  is  of  little  consequence.  A  problem 
arises,  however,  when  some  effectiveness  or  cost  effectiveness  analysis 
task  is  to  be  performed.  Here,  different  categories  of  information, 
which  may  be  stored  in  different  centers,  must  be  obtained.  Moreover, 
the  combination  of  sets  of  data  required  will  generally  vary  from 
analysis  to  analysis.  Hence,  it  is  important  to  determine  whether  the 
required  information  is  available  and  subject  to  retrieval  within 
a  reasonable  time  frame.  The  system  which  is  intended  to  accomplish  this 
function  is  currently  under  development,  and  has  be.n  designated  by  the 
acronym  CQS  (Conmon  Query  System) .  The  purpose  of  this  system  will  be 
to: 


(1)  Interpret  queries  addressed  to  CQS  to  identify  specific  data 
center(s)  that  contain  the  categories  of  information  required, 

(2)  Direct  queries  to  these  identified  centers  in  a  format  acceptable 
to  the  data  center(s)  concerned, 

(3)  Perform  appropriate  searches  and  necessary  manipulations,  and 

(4)  Forecast  results  as  required. 

Initially,  the  principal  application  of  CQS  will  be  directed  towards 
providing  a  data  base  for  performing  various  types  of  effectiveness 
analyses  using  the  GEM  program.  In  addition,  it  is  anticipated  that  CQS 
will  prove  useful  in  problems  which  require  separation  of  data  to  determine 
influences  of  environment,  time  frame,  etc.,  as  well  as  in  applications 
where  combining  of  data  elements  (i.e.,  data  pooling)  is  indicated.  The 
initial  phase  of  this  development  has  been  completed;  being  primarily 
focused  on  a  survey  titled  Evaluation  and  Classification  Study  of  Government 
and  Contractor  Reliability  and  Maintainability  Data  Sources*. 

Optimization  Routines 

Optimization  routines  are  programs  which  evaluate  and  select  from 
alternatives  the  system  which  is  considered  best  in  terms  of  specified 
objectives  and  constraints.  The  art  of  optimization  requires  the  matching 
of  techniques  and  model  forms.  It  appears  that  no  single  optimization 
technique  will  solve  all  problems. 


•Reliability  and  Maintainability  Data  Source  Guide,  SFE  Program, 
NASL  Progress  Report  2,  Lab.  Project  920-72-1,  !fov  1967 
AD-823-228L 
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there  are  sereral  optimization  program*  that  are  not  currently 
part  of  the  GSf  package,  hat  which  could  be  used  when  appropriate. 

(these  techniques  are  highly  specialized  and  should  be  used  only  after 
appropriate  analysis  by  personnel  familiar  with  their  use.)  Descriptions 
and  details,  of  currently  available  computer  programs  in  the  optimization 
area  as  well  as  in  other  systems  performance  effectiveness  areas  such 
as  cost,  maintainability,  logistics,  etc  are  included  in  a  progress  report 
of  the  Haval  Applied  Science  Laboratory  (HASL)*.  In  addition,  a  Department 
of  Defense  report.  Survey  of  Studies  and  Computer  Programming  Efforts  for 
Reliability.  Maintainability  and  Effectiveness.  AD-622  676.  September  1965. 
provides  additional  information  on  these  subjects. 

Output  to  User 

The  normal  output  from  the  GEM  processor  presents  the  results  of 
systems  performance  effectiveness  calculations  in  a  standardized  printed 
format  that  can  be  understood  and  be  useful  to  the  program  office.  Another 
form  of  output  presents  calculated  results  graphically  in  the  form  of  a 
plotted  curve.  The  formats  of  the  graphic  outputs  clearly  indicate  the 
behavior  of  a  systems  performance  effectiveness  variable  as  a  function 
of  time  and  provides  information  necessary  for  decision  making. 


•Evaluation  of  Computer  Programs  for  SR,  HAS7,  Progress  Report  1, 
Lab.  Project  920-72-1,  *ov  1967. 
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For  example,  the  reliability  parameter  can  be  described  by  a  curve  that 
shows  interval  reliability  under  particular  conditions.  The  user  can  then  fit 
his  mission  profile  to  the  curve  and  extract  specific  data  necessary  to  calculate 
effectiveness  as  defined  for  his  system.  Figure  2-8  shows  a  typical  curve.  The 
program  manager  can  specify  particular  intervals  of  interest  and  indicate  the 
policy  and  initial  conditions  to  be  considered  for  each  interval.  , 


At  *  1,  2,  3 . .  24  hours  where  Tq  «  0(A  =  1) 


Probability  of  sustained  performance  for 

At  =  1,  2,  3....»  24  hours  where  T1  =  1  yr.  (A  4  1) 


FIGIIE  M 

FEIFIINANCE  TIME  DEPEKIENCIES 


Summary  of  OEM 

The  techniques  described  in  this  section  constitute  the  current  capability  of 
the  Generalized  Effectiveness  Methodology  (GEM).  OEM  will  give  project  managers 
and  design-and-development  teams  the  capability  to: 

•  Define  system  design  and  generate  analytic  models  at  various  levels 
of  complexity 

•  Specify  changes  in  the  design  or  analytic  models  and  compute  the  effect 
of  certain  changes  on  system  effectiveness  and  sensitivity 

•  Specify  changes  in  specific  design  factors  and  compute  the  effect  on 
other  design  factors 

•  Store  on  tape  complete  details  of  analysed  systems  and  analytic  models. 
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•  Recall,  update  (expand,  modify)  and  reevaluate  any  system  or  analytic 
model  description  stored  In  the  system  library 

•  Expand,  at  will,  the  set  of  computational  equations  stored  In  the 

equation  library 

•  Request  print-outs  of  trade-off  curves  illustrating  the  relationships 
between  any  combination  of  design  factors,  support- system  characteristics, 
and  systems  performance  effectiveness  as  a  function  of  mission  times 

The  following  are  specific  advantages  offered  by  GEM  which  result  in  cost 
savings: 

•  Separate  programs  need  not  be  written  to  evaluate  each  specific  design 
or  analytic-model  configuration. 

•  A  system  can  be  described  directly  in  design-oriented  language  without 
the  need  for  computer  programming. 

•  Specific  changes  in  design  or  analytic  model  can  be  made  without  extensive 
program  revision  or  rewriting  of  programs. 

2.4  PieilEMS  OF  EFFECYIVENESS  ANALYSIS 

To  emphasize  the  fact  that  systems  performance  effectiveness  is  an  envolving 
discipline,  several  of  its  more  important  problems  ere  described  below. 

2.4.1  Mi  salon  Analysis 

Before  analyzing  o  well-defined  system,  let  alone  developing  a  new  one,  it 
is  necessary  to  know  what  the  system  is  supposed  to  do,  i.e.,  what  mission(s) 
it  must  perform.  It  Is  relatively  easy  to  establish  performance  envelopes  for 
various  subsystems  that.  In  turn,  enable  the  system  to  perform  one  task  In  one 
environment.  The  problem,  however,  becomes  much  more  difficult  if  one  must 
consider  many  tasks  (e.g. ,  defend  other  units,  track  a  target  without  getting 
within  vulnerable  range,  or  attack  a  target)  under  many  environments  (calm  sea, 
rough  sea,  or  subzero  weather). 

One  scheme  often  used  is  to  develop  figures  of  merit.  This  scheme  weights 
the  effectiveness  figure  for  each  task  by  the  frequency  with  which  the  system 
may  be  called  upon  to  perform  each  task.  However,  systems  are  developed  to 
satisfy  specific  mission  requirements  that  have  been  formulated  on  the  basis 
cf  specific  threats.  The  best  system  selected  in  accordance  with  such  a  scheme 
may  not,  therefore,  be  capable  of  responding  to  a  specific  threat,  which  of 
itself  may  be  extremely  serious  but  also  may  occur  very  often.  Thus  a  system 
may  not  be  satisfactory  from  the  standpoint  of  a  specific  threat. 

It  is  possible,  however,  to  alter  the  method  by  weighting  the  threats  by 
their  severity  and  expected  frequency  of  occurrence  and  thua  design  the  system 
for  some  other  weighted  average  of  threats.  However,  this  does  not  solve  the 
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problem  either,  since  optimizing  an  answer  to  any  average  threat  does  not  optimize 
the  answer  to  each  threat.  Perhaps  the  solution  is  to  develop  a  procedure  to 
optimize  the  effectiveness  of  answering  a  suitable,  chosen,,  average  threat  that 
is  subject  to  the  achievement  of  a  given  effectiveness  figure  against  each  specific 
threat. 

On  one  hand,  it  makes  little  sense  to  speak  about  the  average  effectiveness, 
the  average  environment,  or  the  average  threat.  On  the  other  hand,  one  cannot 
optimize  with  respect  to  a  single  mission  unless  the  system  has  only  one  task  to 
perform.  Such  a  procedure,  therefore,  leads  to  all  the  difficulties  associated 
with  suboptimization,  which  results  when  one  tries  to  optimize  a  system  by 
optimizing  each  subsystem. 

2.4.2  Figure  of  Merit 

Perhaps  the  difficulty  associated  with  the  formulation  of  an  appropriate 
figure  of  merit  can  be  best  explained  by  a  discussion  of  the  analogous  problem 
encountered  with  the  parameters  of  a  frequency  distribution.  For  example,  with 
a  distribution  a  population  can  be  characterized  by  its  mean;  but  this  does  not 
indicate  the  variability  about  this  mean.  A  measure  of  variability,  called  the 
standard  deviation  can  also  be  added  to  this  characterization. 

Yet  there  will  be  other  properties  of  the  population  that  are  not  pictured 
by  any  of  these  characterizations,  e.g.,  the  lack  of  symmetry.  If  the  10th  and 
90th  percentiles  are  also  given,  however,  a  more  complete  picture  of  the  popula¬ 
tion  begins  to  take  shape.  Nevertheless,  no  finite  set  of  parameters  can  ever 
completely  describe  a  real  population  or  its  frequency  distribution. 

Similarly,  of  one  wants  to  know  the  system-effectiveness  figures  in  all 
situations,  something  analogous  to  the  frequency  distribution  is  needed.  Although 
there  may  be  a  figure  of  merit  analogous  to  the  mean  in  the  frequency -distribution 
example,  it  can  never  give  all  the  information  about  the  system. 

2.4.3  Degraded  Performance 

At  the  component  level,  degradation  can  be  thought  of  as  measured  by  che 
:.^„ber  of  failed  components.  At  the  system  level,  however,  degraded  performance 
may  refer  to  the  probability  of  performing  the  mission.  For  example,  the  system 
might  be  designed  to  be  capable  of  performing  its  mission  95^  nf  the  time  in  one 
environment.  In  another  environment,  however,  it  may  have  only  a  90%  capability, 
and  this  may  be  considered  degraded  performance. 

On  the  other  hand,  consider  a  radar-weapons  system.  Its  design  performance 
consists  of  being  able  to  pick  up  an  object  at  a  certain  distance,  and  ther,  being 
able  to  assign  appropriate  weapons  once  the  object  has  been  correctly  classified. 

A  probability  is  associated  with  each  subsystem  performance,  and  hence  also  with 
the  system's  performance.  If  the  probability  associated  with  any  subsystem  is 
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reduced,  however,  the  probability  of  system  performance  will  also  be  reduced, 
and  this,  too,  can  be  considered  degraded  performance. 

Models  of  degraded  performance  can  also  be  modified  bo  that  they  are  time-  • 

variant.  There  are,  moreover,  many  other  possible  definitions  of  degraded 
performance.  The  one  to  use,  of  course,  is  the  one  that  is  useful  in  assessing 
a  system.  Much  current  work  is  misunderstood  because  one  group  does  not  use  the 
I am  concept  of  system  degradation  as  another. 

2.4.4  Modeling  Problems 

Two  facets  of  modeling  that  have  not  been  referred  to  previously  warrant 
discussion,  since  they  present  significant  problems.  They  concern  assumptions 
and  human  factors. 

Assumptions  of  Independence 

The  most  significant  assumption  made  in  the  modeling  process — and  especially 
in  reliability  and  availability  modeling — is  that  units  comprising  the  system 
fail  and  are  repaired  independently  of  each  other.  In  some  cases  such  an  assump¬ 
tion  may  cause  only  a  slight  distortion  in  simulating  the  functioning  of  the 
system.  On  the  other  hand  such  assumptions  may  so  simplify  the  problem  that  its 
solution,  for  all  practical  purposes,  becomes  trivial. 

Human  Factors 

Navy  concern  for  human  factors  falls  into  three  major  areas:  life  support, 
personnel  and  training,  and  human  engineering.  It  is  the  goal  of  life  support  to 
maintain  and  protect  the  human  by  controlling  his  environment;  the  goal  of  person¬ 
nel  and  training  is  to  select,  train,  and  assign  the  human  for  operational  tasks; 
and  human  engineering  provides  the  design  engineer  with  the  basis  for  the  most 
effective  utilization  of  the  human  component  of  the  system.  In  short  the  discip¬ 
line  of  human  factors  in  systems  performance  effectiveness  requires  that  the  man 
module  be  considered  Just  as  a  hardware  component:  to  be  evaluated  for  cost, 
reliability,  maintainability,  availability,  and  operability.  In  addition,  the 
man  module  must  be  considered  for  trainability. 

To  achieve  these  goals  in  a  disciplined  fashion  certain  procedures  must 
be  followed.  These  procedures  are  not  one-time  events  to  be  accomplished  early 
in  the  development  phase,  but  rather  are  iterative  procedures  that  must  be 
reviewed  and  changed  where  necessary  Just  as  design  is  reviewed  and  changed 
during  an  equipment-  *  a  development. 

Appendix  0  of  Guide  for  Human  Pactora  and  Personnel  Resources  Program. 

NAVMAT  Instruction  4000.20,  Integrated  Logistic  Support  Planning  Procedures, 
contains  an  outline  of  the  procedures  to  be  followed  In  a  human  factors  program, 
and  Hats  a  aeries  of  reference  documents  that  should  be  utilized  In  such  a 
program. 

1  ! 
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The  text  of  NAVMAT  Instruction  4000.20  may  give  the  impression  that  imple¬ 
mentation  of  its  procedures  will  lead  to  the  solution  of  all  Human  Factors  prob¬ 
lems.  Unfortunately,  this  is  not  so.  OEM  has  stressed  and  improved  the  equipment 
aspects  of  the  effectiveness  model,  and  similar  stress  must  now  be  put  on  the 
human  segment  of  the  model.  As  with  hardware  models,  models  for  human  behavior  are 
no  better  than  the  data  inserted  into  them.  Therefore,  data  must  be  collected  which 
has  applicability  to  the  broad  range  of  tasks  and  characteristics  of  the  huaan 
element  of  OEM. 

It  must  be  kept  in  mind  that  the  goal  of  an  equipment-development  project 
is  not  solely  that  of  producing  a  tangible  piece  of  hardware,  but  that  of 
producing  hardware  that  can  be  used  and  maintained  by  men  to  its  fullest  design 
capability. 

2.4.5  Data  Problems 

The  quality  of  the  data  used  to  perform  calculations  during  the  course  of 
a  systems  performance  effectiveness  analysis  will  have  a  significant  effect  on  the 
accuracy  and  utility  of  the  results.  Unfortunately,  the  mathematical  model 
that  describes  a  system  configuration  and  behavior  often  is  far  more  precise  than 
the  input  data  available.  If  effectiveness  values--obtained  from  an  exercise  of 
the  system  mod«ji — are  used  as  relative  rather  than  absolute  values,  the  quality 
of  the  data  is  usually  found  to  be  adequate.  As  a  general  rule,  such  values  are 
satisfactory  when  the  analysis  is  performed  to  obtain  comparisons  between  alter¬ 
nate  designs,  or  to  determine  the  effect  of  changes  on  a  specific  configuration. 

If  absolute  values  are  required,  however,  extreme  care  must  be  used  in  selecting 
the  input  data,  and  caution  should  be  observed  in  interpreting  the  results. 
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SUMMARY  AND  IMPLEMENTATION  ACTIONS 


THREE 


The  systems  performance  effectiveness  discipline  is  a  new  science  which 
still  requires  refinement  by  supporting  research  personnel.  This  manual  has 
described  an  approach  which  allows  program  personnel  to  make  immediate  use  of 
effectiveness  analysis  techniques,  while  improvements  in  data,  system  descriptions, 
forecasting,  and  computer  techniques  are  introduced  in  an  evolutionary  manner. 

Problems  have  been  identified  for  the  following  purposes: 

•  To  emphasize  that  this  manual  presents  a  perspective  of  the  current 
state  of  the  art,  and  not  a  closed  systems  performance  effectiveness 
doctrine 

To  indicate  where  further  investigations  are  required  (and  in  some 
Instances  are  nov  being  carried  out)  and  to  refine  and  solidify  the  art 
of  controlling  and  ng  systeas  performance  effectiveness 

Program  managers,  project  managers,  and  project  engineers  should  take  the 
following  actions  in  implementing  systems  performance  effectiveness: 

Determine  the  underlying  system  performance  effectiveness  factors 
associated  with  the  system  as  discussed  In  Sections  2.1  and  2.2; 
consider  the  life  cycle  status  and  acquisition  planning  as  described  in 
Appendix  A. 
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Develop  a  systems  perfoimance  effectiveness  model  for  the  system  as 
discussed  in  Chapter  1  and  Appendix  B. 

Develop  Integrated  Logistic  Support  aspects  of  the  system  as  shown 
In  Figure  2-3. 

Assure  the  maximum  use  of  technical  communications  In  systems  performance 
effectiveness  activities  as  described  In  Section  2-3- 

Make  use  of  Generalized  Effectiveness  Methodology  (GEM)  for  systems 
performance  effectiveness  analysis  as  described  in  Section  2.3-  (Consul¬ 
tation  may  be  required  for  the  application  of  GEM--discussion  with  the 
U.  S.  Naval  Applied  Science  Laboratory  is  recommended.) 


Careful  performance  of  these  actions  will  assure  acquisition  of  a  system 
whose  systems  performance  effectiveness  characteristics  are  consistent  with  its 
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APPENDIX  A 


THE  RELATION  OF  SYSTEMS  PERFORMANCE  EFFECTIVENESS  TECHNIQUES 
TO  THE  RDT&E  PLANNING  AND  ACQUISITION  PROCESS 

1.  The  Steps  in  System  Acquisition 
1.1  Introduction 

The  system-acquisition  process  begins  with  the  generation  of  a  General 
Operating  Requirement  that  defines  a  need  for  capability  in  a  functional  Naval 
warfare  area.  It  progresses  through  Concept  Formulation,  Contract  Definition, 
Engineering  or  Operational  Development,  and  Production,  and  it  terminates  with 
the  procurement  of  the  last  replacement  part  for  the  system  that  has  been 
developed  and  produced  to  fulfill  the  need. 


DOD  Directive  3200. 9,  1  July  1965*  "Initiation  of  Engineering  and  Operational 
Svstem  Development,"  implemented  by  SECNAV  Instruction  3900.33,  20  August  1965, 
establishes  the  phases  of  Concept  Formulation  (CF)  and  Contract  Definition  (CD) 
as  prerequislties  to  undertaking  full  Engineering  Development  of  major  projects. 
(These  phases  replace  the  Project  Definition  Phase  of  DOD  Directive  3200.9, 

26  February  1964. )  As  shown  in  Figure  A-l,  Concept  Formulation,  which  comes 
under  the  general  classification  of  Exploratory  or  Advanced  Development,  includes 
the  RDT&E  Planning  Dialogue  between  user  and  producer.  Successful  completion  of 
Concept  Formulation  leads  to  Contract  Definition,  the  first  step  in  Engineering 
Development.  In  CD,  generally,  two  or  more  competitive  contractors,  working 
closely  with  the  Navy,  develop  concept,  design  approach,  trade-off  solutions, 
management  plans,  schedule,  and  overall  cost.  Satisfactory  completion  of  CD  is 
followed  by  System  Development,  Production,  Installation,  and  Operation. 
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Figure  A-2  lists  the  governing  OFNAV,  SECNAV,  and  OS  directives  for  the 
steps  in  System  Development,  together  with  the  Navy  manuals  developed  to  give 
guidance  in  the  preparation  of  RBT&E  planning  documentation. 

The  actual  system-acquisition  process  is  not  always  as  orderly  as  described 
above.  Projects  may  leap-frog  over  Contract  Definition,  or  they  may  retrogress 
from  Engineering  Development  back  to  Advanced  or  Exploratory  Development.  Improve¬ 
ment  programs  for  Fleet  operational  systems  may  appear  to  start  life  in  the  System 
Development  Phase.  It  is  important  to  realize,  however,  that  although  the  formal 
acquisition-process  terminology  for  the  steps  in  system  development  may  not  always 
exist,  these  steps  do  take  place  in  almost  every  program. 
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*  NAVMAT  P3910,  "Guide  for  the  Preparation  of  Technical  Development 
Plans",  July  1965 

**  NAVMAT  P-3910A  and  Supplement  1,  "Guide  for  the  Preparation  of 
Proposed  Technical  Approaches  (PTA)",  February  1966 


1.2  The  RDT&E  Planning  Dialogue 


Concept  Formulation  begins  with  the  RDT&E  (Research,  Development,  Test, 
and  Evaluation)  Planning  Dialogue,  as  defined  in  the  3910  series  of  OPNAV 
Instructions  and  the  3200  series  of  DOD  Directives. 

RDT&E  planning  within  the  Department  of  the  Navy  is  characteristically 
conducted  as  a  dialogue  between  the  user  interest  and  the  producer  interest.  The 
user  interest  is  represented  by  the  Chief  of  Naval  Operations  and  the  Commandant 
of  the  Marine  Corps,  as  spokesmen  for  the  operational  forces;  and  the  producer 
Interest  is  represented  by  the  Chief  of  Naval  Material  speaking  for  the  Naval 
Material  Command.  This  user-producer  relationship  is  more  analogous  to  a 
relationship  between  cooperating  Independent  business  organizations  than  to 
traditional  military  relationships.  Plans  are  the  result  of  negotiation  between 
the  two  interests.  Through  this  process  trade-offs  are  made  that  will  result  in 
the  maximum  military  capability  for  the  operating  forces  within  the  limits  of  the 
resources  available. 

The  principal  documents  used  in  the  user-producer  dialogue  are  shown  in 
Figure  A- 2  as  the  intermediate  points  in  the  flow  diagram.  At  first  glance  the 
impression  is  that  the  user  interest  levies  unilateral  requirements — based  on 
pure  military  necessity — on  the  producer  interest.  The  actual  process,  however, 
involves  a  continuous  interaction  between  operational  requirements  and  their 
spokesman,  and  technical  and  scientific  possibilities  and.  their  spokesman.  It  is 
a  continuing.  Iterative  interchange .  New  formal  requirements  for  weapons  hard¬ 
ware  often  have  their  genesis  in  new  possibilities  stemming  from  advancing 
knowledge  and  technology  rather  than  from  evolving  military  need  or  reassessment 
of  old  needs. 

The  Chief  of  Naval  Operations  is  responsible  for  the  preparation  of  a 
General  Operational  Requirement  (GOR)  for  each  functional  warfare  and  support 
area.  GOR's  usually  result  from  rather  extensive  long-range  strategic  and  tacti¬ 
cal  studies.  These  documents  state,  In  relatively  broad  but  significant  terms, 
the  capabilities  the  Navy  needs  within  each  area.  For  guidance  In  making  trade¬ 
offs  in  weapons  design,  the  GOR  should  indicate  tne  relative  Importance  of  the 
needed  capabilities.  In  the  past,  performance  capabilities  have  been  adequately 
stated  In  the  GOR’s;  however,  other  considerations  that  constitute  system 
effectiveness — reliability,  maintainability,  etc , — have  not  always  been  given 
adequate  attention.  Systems  perfoimance  effectiveness  guidance  must  be  provided 
for  the  entire  system  at  the  GOR  stage,  for  here  is  where  the  thinking  and  planning 
for  cotal  system  effectiveness  begins. 

In  some  cases  the  using  agency  Issues  a  document  concerning  a  narrower  require¬ 
ment,  a  Tentative  Specific  Operational  Requirement  (TSOR).  This  document  states 
the  need  for  achieving  a  particular  operational  capability  and  outlines  the  identi¬ 
fiable  system  characteristics  necessary  to  fulfill  the  requirement.  The  TSOR 


defines  the  desired  performance  goals  and  provides  additional  information  needed 
to  weigh  alternatives  and  make  the  trade-offs  required  for  an  optimum  system. 

The  producer  response  to  either  the  GOR  or  TSOR  is  a  PTA  (Proposed  Technical 
Approach).  PTA's  are  developed  by  the  Naval  Material  Command  to  propose  tech¬ 
nically  feasible  alternative  methods  of  accomplishing  objectives  set  forth  in 
a  GOR  or  TSOR.  The  FTA  should  be  fully  responsive  to  the  GOR  or  TSOR;  therefore, 
the  quality  of  the  PTA  depends  directly  on  the  quality  of  the  GOR  or  TSOR.  In 
addition  to  other  mandatory  requirements  of  the  PTA,  the  governing  OPNAV  and  DOD 
directives  require  that  the  FTA  analyze  and  compare  the  operational  effectiveness 
of  the  proposed  alternate  development  approaches  in  terms  of  performance,  relia¬ 
bility,  operability,  and  maintainability,  and  clearly  indicate  the  basis  of  the 
comparison,  such  as  previous  experiments,  extrapolation,  or  conjecture. 

The  user  reviews  what  is  presented  in  the  PTA  and  decides  on  one  of  the 
following  alternatives; 

1.  Study  the  requirement  further 

2.  Begin  feasibility  studies,  further  experimentation,  or  both 

3.  Begin  an  engineering  or  operational  development  effort 

4.  Terminate  development  effort  in  the  specific  area 

If  alternative  1  is  chosen,  the  process  returns  to  the  strategic  and 
tactical  study  phase  and  usually  results  in  revisions  to  the  GOR  or  TSOR.  If 
alternative  2  is  chosen, the  user  interests  develop  and  promulgate  an  Advanced 
Development  Objective  (ADO) .  If  alternative  3  is  chosen,  the  user  Interests 
develop  and  promulage  a  Specific  Operational  Requirement  (SOR) .  In  the  case  of 
alternative  4,  all  effort  proposed  in  the  PTA  is  terminated,  which  usually  results 
in  the  action  indicated  for  alternative  1,  although  on  occasion  the  requirement 
will  remain  unmodified  and  essentially  dormant  until  research  effort  develops  new 
te  hnlcal  approaches  to  be  Incorporated  in  a  superseding  PTA 

If  alternative  2  (ADO)  or  alternative  3  (SOR)  is  chosen,  the  producer 
prepares  a  Technical  Development  Plan  (TDP).  However,  there  is  a  distinct 
difference  between  a  TDP  that  responds  to  an  ADO  and  one  that  responds  to  an 
SOR.  In  the  case  of  an  ADO,  the  effort  defined  by  the  TDP  is  either  directed 
toward  demonstration  of  feasibility  of  approaches )  or  experimentation  at  the 
breadboard  level.  This  effort,  if  successful,  leads  to  an  SOR  and  a  responding 
TDP. 


The  TDP  responding  to  an  SOR  represents  the  essential  completion  of  Concept 
Formulation  (CF).  The  most  important  end  product  of  CF,  it  comprises  the  complete 
and  detailed  plan  for  fulfilling  the  operational  requirements  of  the  user.  The 
goal  of  a  TDP  is  a  balanced  and  Integrated  effort  for  optimizing  operational 
effectiveness,  total  cost,  and  early  availability. 
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With  the  development  of  the  TDP,  the  necessary  RDT&E  Planning  for  the 
full-scale  development  phase  of  the  system  is  established;  if  planning 
has  been  adequate,  only  a  minimum  of  TDP  updating  will  be  required  during  the 
full-scale  development  phase. 

2.  Systems  Performance  Effectiveness  and  the  System-Acquisition  Process 

2.1  The  Discipline  of  Systems  Performance  Effectiveness  Engineering 

The  role  of  systems  performance  effectiveness  measures  during  the  acquisition 
process  is  to  enable  the  program  manager  to  restructure  the  allocated  goals  to 
the  system  level,  thereby  allowing  system  decisions  to  be  made  by  higher  management. 
The  allocation  process  is  shown  in  Figure  A-3,  this  ability  adds  a  new  dimension 
to  management.  Dynamic  life  cycle  management  and  Integrated  Logistics  Support 
then  become  practical  goals.  The  above  structuring  process  is  the  systems 
performance  effectiveness  model,  frequently  discussed  but  too  often  not  used 
until  after  the  fact.  Effectiveness  models  are  described  in  Appendix  B. 

The  following  sections  describe  the  application  of  the  Systems  Performance 
Effectiveness  discipline  to  the  System  Acquisition  process.  It  will  be  seen  that 
systems  performance  effectiveness  techniques  are  intended  to  aid  the  project 
manager  in  decision-making  by  presenting  him  with  organized  Information,  and  to 
assist  him  in  assigning  task  priorities  by  highlighting  critical  areas  within 
his  project.  The  discipline  of  Systems  Performance  Effectiveness  Is  not  a 
replacement  for  managerial  Judgment;  rather,  it  supplies  a  basis  for  better  and 
more  timely  decisions. 


r77TT7T777777T77777777777777m 

EFFECTIVENESS  DESCRIPTION  * 


Resources 


Sub  Set  of  P„ 


0 


Sub  Set  of  PT  Initial  Constraints 


o 


Q 


Allocation  of 
Goals  4  Requirements 


Allocation 


Allocation 


fi¬ 


rm  mii  nil 

I  HORIZONTAL  MANAGEMENT  (Surveillance  4  Test)  (Z 


-N 


-V 


FIMtE  A  3 

ALLOCATE  HICttS  FN  LAKE  SYSTEMS 


A-7 


2.2  Concept  Fonnulatlon 


"Concept  Formulation  describes  the  activities  preceding  a  decision  to  carry 
out  Engineering  Development.  These  activities  include  accomplishment  of  compre¬ 
hensive  system  studies  and  experimental  hardware  efforts  under  Exploratory  and 
Advanced  Development  and  are  prerequisite  to  a  decision  to  carry  out  Engineering 
Development . "* 

The  initiation  of  successful  system  development  programs  in  the  U.  S.  Navy 
is  becoming  Increasingly  difficult  in  view  of  the  rapid  technological  changes  to 
be  coped  with  and  the  growth  of  required  program  documentation.  Success  depends 
upon  many  complex  factors  such  as  the  following: 

•  Determination  of  threat  profiles  and  their  translation  into  system 
requirements  and  constraints 

•  Status  and  understanding  of  performance  parameters,  resource  estimates, 
and  error  budgets  of  exploratory /advanced  development  projects 

•  Understanding  of  the  activities  required  to  satisfy  the  directives, 
requirements,  and  instructions  of  the  DOD/Navy  management  system 

•  Availability  of  people,  facilities,  techniques,  and  data  to  support 
required  activities 

The  integration  of  the  above  factors,  and  others,  for  the  specific  purpose 
of  initiating  an  engineering  development  program  is  known  as  Concept  Formulation. 

By  definition  then,  Concept  Formulation  does  not  replace  existing  analyses  or 
development  activities  but  rather  seeks  to  optimally  integrate  the  outputs  of 
such  activities.  Much  work  has  to  be  done  to  provide  efficient  concept-formula¬ 
tion  capability.  A  cohesive  marriage  of  the  design  and  analysis  techniques  is 
required.  The  results  of  concept-formulation  studies  will  have  a  major  impact 
upon  the  cost  and  responsiveness  of  future  Naval  systems. 

2.2.1  Candidate-System  Definition 

In  an  ideal  situation  the  process  of  Concept  Formulation  would  progress  from 
the  recognition  of  a  threat,  to  a  number  of  approaches,  to  candidate  concepts,  and 
then  to  candidate  systems.  This  idealised  order  Is  seldom  realized.  The  lack  of 
orderliness  in  the  real-world  evolution  of  the  process  can  pose  extreme  problems 
if  not  approached  in  a  disciplined  manner. 

Basic  to  a  disciplined  approach  is  the  recognition  that  these  stages  of 
progress  are  simp1.;'  differing  degrees  of  precision  In  defining  the  system.  In 
other  words,  the  descriptive  parameters  become  progressively  better  defined  for  each 
step  in  the  evolution  from  approach  to  candidate  system. 

During  the  earlier  stages,  system  functions  are  quite  broadly  defined.  The 
gross  functions  are  progressively  structured  as  groups  of  subfunctions,  each  with 

*DOD  Directive  3200.9  of  1  July  1965 
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its  associated  Inputs  and  outputs.  Structuring  continues  until  the  candidate 
system  has  been  defined.  This  approach  permits  comparative  evaluation  of  com¬ 
peting  candidate  systems,  regardless  of  their  relative  stage  of  evolution. 

2.2.2  Preferred-System  Selection 

The  preferred-system  selection  process  must  take  Into  account  considerations 
other  than  systems  performance  effectiveness.  Among  these  are  coBt-effeetiveness 
and,  in  the  Navy  System  Effectiveness/Cost-Effectiveness  (SE/CE)  method,  what 
has  been  termed  Defense  Effectiveness. 

The  cost-effectiveness  analysis  is  based  on  two  cost  estimates  associated 
with  each  function  in  the  systems  performance  effectiveness  model.  One  estimate 
covers  the  cost  of  acquisition;  the  other,  covers  the  cost  of  utilization  or 
ownership.  The  former  includes  the  RDT&E  costs,  prorated  over  the  anticipated 
production  quantity,  as  well  as  the  production  (and  installation,  when  appropri¬ 
ate)  cost  per  system.  The  latter  includes  all  operating,  maintenance,  and  support 
costs  of  the  system.  These  costs  can  be  used  in  connection  with  trade-off  analyses, 
or  they  can  be  aggregated  and  associated  with  the  systems  performance  effectiveness 
index  and  used  as  partial  determinants  in  preferred-system  selection.  Other 
partial  determinants  useful  in  preferred-system  selection  are  comparisons  of 
systems  performance  effectiveness  indexes  with  manpower  and  lead-time  requirements. 

The  results  of  the  effort  to  this  point  are  formally  organized  as  a  PTA  and 
submitted  through  appropriate  channels  to  the  Chief  of  Naval  Operations  for  com¬ 
pletion  of  Preferred  System  Selection. 

The  Defense  Effectiveness  methodology  is  based  on  the  evaluation  of  military 
worth  and  its  degradation  as  a  function  of  the  time  taken  to  acquire  the  system. 
Neither  of  these  factors  is  directly  measurable.  However,  they  can  be  assigned 
numeric  Judgment  valuations  by  military  experts.  The  military- worth  index  is 
an  evaluation  of  the  mission  to  be  accomplished  by  the  system.  If  all  candidate 
systems  accomplish  identical  missions,  the  military-worth  valuation  can  be  set 
at  unity,  and  only  the  effect  of  time-to-acquire  will  be  considered  in  the  eval¬ 
uation.  On  the  other  hand,  if  one  or  more  of  the  candidate  systems  are  capable 
of  accomplishing  additional  missions  or  different  combinations  of  missions,  the 
indexes  of  military  worth  should  reflect  the  differences  as  military  Judgment 
may  deem  appropriate.  The  net  effect  of  the  Defense  Effectiveness  methodology 
is  to  provide  a  military- Judgment  coefficient  to  assist  in  Preferred  System 
Selection. 

The  actual  selection  is  suggested  by  the  candidate  system  with  the  highest 
index  of  Defense  Effectiveness.  However,  this  suggestion  is  not  absolute. 

Modeling  assists  in  the  decla ion-making  process  and  is  not  a  substitute  for 
managerial  Judgment.  Indeed,  Judgment  may  result  in  a  decision  at  this  cfc age 
that  more  than  one  preferred  system  will  enter  Contract  Definition. 
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The  final  action  in  the  process  of  Preferred  System  Selection  is  taken  by 
the  Chief  of  Naval  Operations.  He  expresses  the  decision  through  the  issuance 
of  an  ADO  or  an  SOR.  If  more  than  one  Pi*eferred  System  is  indicated,  an  ADO  is 
issued.  (It  should  be  noted  that  this  is  but  one  circumstance  under  which  /an 
ADO  may  be  issued).  If  a  single  preferred  system  is  indicated,  an  SOR  is  issued. 

2.2.3  Approval  to  Enter  Contract  Definition 

The  preferred  system(s)  having  been  selected,  one  step  remains  prior  to  Con¬ 
tract  Definition,  To  the  project  manager,  this  is  one  of  the  most  critical  steps, 
and  his  first  major  test  as  a  manager.  He  must  demonstrate  that  he  has  met  all 
of  the  prerequisites*  to  obtain  SECDEF  approval  to  enter  into  Contract  Definition. 

If  not  approached  In  a  well-organised  Manner,  this  demonstration  can  be 
a  time  -  consuming  and  frustrating  exercise.  However,  the  Navy  SE/CE 
methodology,  with  its  models  (including  the  system  model)  provides  the 
ordered  approach  and  the  demonstration  vehicle.  Using  these  models,  the 
manager  can  define  the  preferred  syetem(s)  in  terms  of  technical  goals  and 
criteria,  trade-off  evaluations,  and  priorities  of  effort,  together  with 
the  associated  confidence  levels. 


Application  of  the  Navy  SE/CE  methodology  throughout  the  Concept  Formulation 
phase  of  the  system's  life  cycle  places  the  manager  in  an  unambiguous  position. 

If  he  can  define  his  preferred  system  sufficiently  well  to  exercise  the  models, 
it  is  probable  that  his  system  is  soundly  conceived  and  that  the  model  completion 
in  itself  will  demonstrate  his  meeting  of  the  prerequisites  for  Contract  Defini¬ 
tion.  On  the  other  hand,  inability  to  provide  minimal  input  requirements  for 
model  analysis  and/or  to  provide  clear  preferred-system  definition  is  a  strong 
indication  that  prerequisities  have  not  been  met. 


Having  successfully  demonstrated  accomplishment  of  prerequisites  through 
Preferred  System  Definition,  the  manager  uses  the  essential  inputs  to  prepare  the 
Request  for  Proposal  (RFP)  needed  to  cover  the  contracted  effort. 


2.3  Contract  Definition 

Contract  Definition  is  a  period  of  major  concern  to  the  project  manager, 
although  the  burden  of  proving  performance  ar.d  responsiveness  rests  on  the 
contractor  (pr_vate  industry  or  Government  ^laboratory)  who  has  beer,  selected 
a  qualified  participant  essentially  on  the  basis  of  proposals. 


•OPNAV  Publication  ^0  P-1,  11  June  19o5. 
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In  many  respects  the  application  of  the  Navy  SE/CE  methodology  to  Contract 
Definition  parallels  its  application  to  Concept  Formulation.  There  are,  however, 
some  significant  differences.  In  time  sequence,  the  application  of  the  Navy 
SE/CE  methodology  under  Concept  Formulation,  as  discussed  in  Section  2.2,  is  appli¬ 
cable  if  the  term  "Contractor's  Proposed  System"  is  used  in  lieu  of  "Candidate 
System. 

The  preferred  system(s)  having  been  defined  in  the  last  step  in  Concept 
Formulation,  a  sensitivity  analysis  is  performed  with  the  effectiveness  model . 

This  analysis  will  indicate  the  limiting  parameters  and  priorities  for  each  ele¬ 
ment  of  the  system  model,  which  are  expressed  in  terms  of  technical  goals  or 
requirements.  The  range  of  the  pennissive  parameters,  properly  related  to  esti¬ 
mates  of  state-of-the-art  capabilities,  establishes  the  degree  of  criticality 
of  the  element. 

The  preferred-system(s )  definition  and  the  critical  systems  performance 
effectiveness  parameters  are  incorporated  into  a  Request  for  Proposal  (RFP), 
which  is  transmitted  to  the  contractor(s)  as  a  guide  for  proposed  approaches  to 
Contract  Definition.  The  preferred-system(s)  definition  provides  for  Phase  A  of 
Contract  Definition,  the  equivalent  of  Candidate  System  Definition  in  Concept 
Formulation.  In  addition  to  guidance  for  the  contractor (s),  the  definition  of 
critical  systems  performance  effectiveness  parameters  provides  the  criteria  for 
evaluating  contractor  proposals  under  Phase  A. 

As  with  other  aspects  of  the  systems  performance  effectlvenss  discipline, 
the  definition  of  critical  systems  performance  effectiveness  parameters  is  not 
static.  The  process  of  refinement  started  in  Concept  Formulation  continues  in 
Contract  Definition.  As  a  result  of  the  analysis  of  proposals  received  under 
Phase  A  of  Contract  Definition,  the  preferred-system(s)  definition  is  refined, 
and  the  critical  parameters  are  better  defined  by  the  Inputs  received  from  the 
responses  to  the  RFP.  This  sharpening  becomes  most  important  to  the  project 
manager  during  P‘  -se  C  of  Contract  Definition. 

The  project  manager  exercises  little  corral  over  the  Contract  Definition 
effort,  which  is  largely  in  Phase  B.  However,  progress  reports  under  the  contracts 
for  Phase  B  do  provide  definition  and  validating  data.  As  these  data  are  received, 
reiterative  exercise  of  the  systems  performance  effectiveness  model  provides  a 
significant  measure  of  the  progress  being  realized. 

While  the  critical  systems  performance  effectiveness  parameters  can  be 
defined  initially  during  the  early  phases  of  Concept  Fcrmulation,  they  reach 
their  full  definition  during  the  latter  stages  of  Phase  B  and  during  the  analysis 
effort  under  Phase  C  of  Contract  Definition.  They  provide  the  essential  framework 
for  the  decision  to  enter  into  Engineering  Development. 
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The  definition  of  these  parameters  at  this  point  in  the  evolutionary  cycle  of 
a  system  must  be  sharpened  to  the  point /where  the  project  manager  can  demonstrate 
the  following  within  an  18-week  period: 

•  The  operational  goals  and  technical  goals  are  in  agreement 

•  The  technical,  and  hence  operational,  criteria  can  be  met 

•  The  financial  and  schedule  f  otors  are  credible 

•  The  development  risks  are  acceptable 

•  A  defiiiitive  contract  can  be  entered  into  with  the  best-qualified 

contractor  , 

To  demonstrate  the  foregoing,  not  only  must  the  parameters  be  clearly  and 
concisely  defined,  but  they  must  be  quantitatively  interrelated.  This  requires 
highly  structured  system  models  in  ter  '.s  of  functional  block  diagrams  with 
associated  characteristics  values  (or,  in  some  cases  computer  simulation  models) 
and  a  completely  structured  SK/GE  model  with  which  to  analyze  and  evaluate  the 
system  models.  The  former  is  an  output  of  Phase  B  contractor  efforts  The 
latter,  however,  is  largely  the  result  of  the  efforts  of  the  project  manager's 
staff. 


The  success  or  failure  of  Phase  C  will  be  determined  ’ey  the  degree  of  com¬ 
pleteness  of  the  model  and  the  decree  to  which  its  structuring  conforms  to  the 
real-world  situation. 

If  the  SE/CE  model  does  approximate  reality  suet  ssfully,  the  parameters  can 
be  interrelated,  and  the  exercise  of  the  model  lor  each  of  the  competing  systems 
produced  by  the  CD  contractors  provides  a  framework  for  Source  Selection  and  demon¬ 
strates  the  validity  of  entering  into  Engineering  Development  of  the  project,  con¬ 
tinuing  with  further  definition  or  advanced  development  effort,  or  abandoning  the 
project.  ’ 

In  addition  to  its  use  as  a  decision-making  tool,  the  SE/CE  mcctel  also  serves 
another  function  during  this  period.  The  sharp.'  defined  Critical  Effectiveness 
Parameters  provide  within  the  SE/CE  model  the  checklist  for  completing  the  speci¬ 
fication  for  the  Engineering  Development  contract.  This  is  particularly  Important 
in  that  one  of  the  principal  objectives  of  the  Contract  Defjnitton  process  Is  to 
assure  that  a  complete  and  unambiguous  specification  (definition)  is  developed  for 
the  Engineering  Development  effort. 


p . 4  Engineering  Development 

Through  the  process  of  Contract  Definition,  tne  project  manager  has  been 
establishing  a  frame  of  reference  to  define  the  system,  its  technical  goals  -<<nd 
criteria,  and  the  measures  by  which  Its  effectiveness  In  terms  of  its  mission 
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life  costs  can  be  evaluated.  Having  established  this  frame  of  reference,  he  must 
noi.  address  himself  to  obtaining  assurance  of  achieving  an  effective  system. 

The  ultimate  evaluation  of  the  Engineering  Development  phase  occurs  during 
the  test  and  evaluation  of  the  developed  system.  If  the  simulated  system  model 
and  systems  performance  effectiveness  analytic  model  are  valid  and  adequately 
defined,  the  system  should  meet  its  test  and  evaluation  successfully,  and  the 
project  manager  will  have  been  successful. 

If  the  system  is  not  satisfactory,  the  models  have  yet  another  function. 

The  operational  data  accumulated  during  T&E  should  be  inserted  in  the  models. 

The  models  should  then  be  exercised  and  the  results  analyzed  to  identify  problem 
areas.  These  should  then  be  recorded  and  made  available  to  other  project  managers 
to  assist  them  in  avoiding  similar  errors.  At  the  same  time,  a  closed-lfiop  manage¬ 
ment  system  should  be  implemented  to  correct  the  problems. 

If  the  project  is  to  be  continued,  whether  or  not  the  T&E  is  successful,  the 
T&E  data  are  inserted  in  the  models  to  sharpen  further  the  definition  of  technical 
goals  and  criteria  and  to  validate  the  data  for  the  production  baseline  and  pro¬ 
duction  specification.  Here,  again,  the  models  serve  to  guide  the  effort  and  to 
assure  the  project  manager  that  the  baseline  (specification)  is  complete  and 
defined  as  sharply  as  the  aggregate  experience  will  permit.  This  is  a  necessary 
exercise,  whether  or  not  the  R&D  contractor  is  also  the  initial  production  con¬ 
tractor. 

At  this  point,  the  project  manager  finds  himself  subjected  to  conflicting 
pressures.  The  extreme  of  one  school  of  thought  might  insist  that  the  design  be 
frozen  and  production  should  proceed  on  essentially  an  exact-copy  basis.  The 
other  extreme  is  to  continue  injecting  all  of  the  latest  state-of-the-art  improve¬ 
ments  into  production  to  ensure  the  maximum  possible  capabilities  in  the  operating 
forces.  There  are  too  many  factors  external  to  the  system  itself  to  justify  either 
of  these  positions. 

However,  two  prerequisites  to  any  decision  are  apparent.  First,  the  project 
manager  must  have,  with  relative  exactness,  a  complete  definition  of  his  system 
to  serve  as  a  baseline  for  a  decision.  Second,  he  must  have  a  means  for  evaluating 
the  relative  Impacts  of  the  alternatives.  The  system  model  provides  the  former, 
and  the  analytic  model  provides  the  latter. 

2.5  Production 

When  the  system  has  passed  the  test  and  evaluation  phase  and  has  been 
approved  for  service  use,  the  project  manager  must  produce  the  system  and  intro¬ 
duce  it  into  the  operational  forces.  In  the  past,  this  transition  from  Research 
and  Development  to  Production  ha3  meant  turning  the  project  over  to  a  new  team, 
all  too  frequently  Involving  a  great  deal  of  learning  for  the  new  team,  time 
losses,  and  a  loss  of  experience  and  data. 
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'iio  factors  could  provide  safeguards  against  these  traditional  difficulties. 
The  first,  the  project-manager  concept,  includes  provisions  for  keeping  the  manage¬ 
ment  team  intact.  The  same  management  team  that  was  responsible  for  R&D  should 
have  some  continuing  responsibility  for  production.  Thus  the  time  loss  involved 
in  learning  the  system  is  eliminated.  The  second,  use  of  both  simulation  and 
analytic  models,  provides  a  methodology  for  experience  and  data  retention.  The 
formal  structuring  and  recording  of  data  provide  a  high  degree  of  assurance  that 
both  experience  and  data  will  be  retained. 

When  viewed  objectively,  the  demands  placed  on  the  projeot  manager  for 
changes  in  configuration,  cost,  and  schedule  differ  little  in  concept  from  the 
trade-off  analyses  performed  during  concept  formulation  or  the  preferred-system 
selection  performed  during  Contract  Definition.  Indeed,  the  same  tools,  the 
simulation  model  and  the  analytic  model,  can  be  used.  Actually,  since  the  model 
values  have  now  been  more  sharply  defined  through  the  introduction  of  experimental 
data  during  Contract  Definition,  Engineering  Development,  and  Test  and  Evaluation, 
the  validity  of  the  models  as  decision-making  aids  should  be  very  high. 

2 . 6  Fleet  Operations 

The  mo.st  critical  aspect  of  systems  performance  effectiveness  during  Fleet 
operations  is  the  retrieval  of  data  for  future  programs.  One  significant  attempt 
to  provide  a  portion  of  these  data  from  operating  units  Is  the  MDCS  (Maintenance 
Data  Collection  System)  carried  cut  under  the  Navy  3M  Program*.  Attempts  are 
being  made  to  structure  the  MDCS  data  formats  in  such  a  way  that  the  requisite 
inputs  to  the  systems  performance  effectiveness  methodology  can  be  obtained  from 
the  MDCS  without  additional  reporting  requirements.  If  these  efforts  are  success¬ 
ful,  the  project  manager  will  have  available  to  him  the  main  body  of  experimental 
data,  which  can  then  be  Introduced  into  the  models. 

These  data  are  needed  for  two  principal  reasons: 

(1)  They  provide  the  real-life  validating  information  on  all  of  the  project 
manager's  past  decisions.  Through  this  validation  effort  he  can 
determine  the  adequacy  of  weighting  and  other  judgment  factors  that 
were  applied  during  the  preceding  phases.  An  added  return  is  the 
recording  and  sharing  of  these  evaluations  with  project  managers  for 
other  systems  under  development  or  for  superseding  systems.  In  this 
application  of  the  systems  performance  effectiveness  methodology,  the 
Navy  can  receive  substantial  benefits  in  experience  retention. 

(2)  These  data  can  also  be  used  to  establish  a  decision  baseline  for 
determining  the  need  for  so-called  product  improvements  in  operating 
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systems,  and  for  evaluating  proposed  changes.  Costly,  changes  and 
changes  of  questionable  return  may  result  from  use  of.  inadequate  or 
incomplete  data. 

In  the  operational  phase,  as  in  the  preceding  phases,  the  discipline  of  the 
systems  performance  effectiveness  methodology  guards  against  making  decisions  on 
the  basis  of  inadequate,  incomplete,  or  unrelated  data — principally  through  the 
visibility  that  the  modeling  techniques  give  to  the  ramifications  of  the  variances 
in  data  inputs.  While  the  systems  performance  effectiveness  methodology  is  by  no 
means  a  panacea  and  certainly  not  a  substitute  for  sound  judgment.  It  does  provide 
a  structured  discipline  that  substantially  increases  the  probability  that  the 
project  manager  will  have  as  inputs  to  his  deliberations  all  the  factors  necessary 
to  assure  that  he  makes  the  right  decisions. 
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CONSIDERATIONS  FOR  SYSTEMS  PERFORMANCE 
EFFECTIVENESS  MODELS 

1.  Analytic  Models  for  Effectiveness  Evaluation 

Any  meaningful  application  of  the  systems  performance  effectiveness  concept 
to  a  particular  project  requires  a  quantitative  methodology  to  evaluate  the 
effectiveness  of  a  proposed  or  actual  system  in  terms  of  selected  measures, 
requirements,  and  decision  criteria.  Until  this  is  done,  the  concept  of  systems 
performance  effectiveness  for  a  project  has  little  use — except  perhaps  as  a 
rallying  point  for  arguments  about  the  advantages  of  System  X  over  System  Y. 

The  need  for  a  systems  performance  effectiveness  evaluation  methodology  begins 
at  the  inception  of  the  system  life  cycle  and  continues  through  the  succeeding 
design,  development,  production,  test,  and  operational  phases.  Despite  the 
obvious  differences  in  the  depth  of  the  analysis  applicable  to  these  phases,  the 
need  for  a  quantitative  methodology  applies  throughout. 

Evaluation  methodologies  for  systems  performance  effectiveness  characteristics 
can  be  broadly  characterized  in  terms  of  two  approaches:  the  empirical  and  the 
analytic. 

An  empirical  methodology  is  one  devoted  to  data  collection  and  evaluation 
of  existing  systems.  Thus  it  is  possible  to  evaluate  systems  performance  effec¬ 
tiveness  by  means  of  performance  observations  of  systems  in  the  field.  While 
this  approach  is  undoubtedly  the  most  accurate,  it  is  feasible  only  for  systems 
or  projects  that  are  very  far  advanced  in  their  life  cycles. 

An  analytic  methodology,  on  the  other  hand,  is  one  that  derives  its  results 
by  inference,  and  uses  a  set  of  assumptions  and  procedures  as  a  framework  to 
compute  an  effectiveness  description  of  the  system  in  question.  This  descriptive 
system  framework  is  called  an  analytic  model,  and  the  description  of  system  "X" 
in  these  terms  is  called  the  analytic  model  of  system  "X". 

Purely  empirical  or  purely  analytic  methodologies  are,  of  course,  not  very 
useful.  The  former  yields  highly  authoritative  data  too  late  to  be  useful,  while 
the  latter  yields  answers  unsupported  by  factB.  In  practice,  a  balance  is  sought. 
This  balance  will  normally  change  during  the  life  cycle  of  a  system.  As  data 
about  the  behavior  of  the  system  become  more  available,  the  analytic  model 
gradually  merges  into  an  empirical  model;  as  the  data  become  more  available  and 
as  confidence  in  their  value  increases,  statistical  sample  data  supplant  the 
assumptions. 
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Analytic  models,  moreover,  usually  remain  useful  even  with  regard  to  the 
empirical  data  obtained  from  system  samples  taken  during  acceptance  tests.  Also, 
these  data  often  require  interpretation  simply  because  it  is  impractical  to 
conduct  tests  that  are  sufficiently  elaborate  to  yield  statistically  significant 
effectiveness  data  directly. 

The  need  for  analytic  models  to  predict  systems  performance  effectiveness 
thus  emerges  from  the  need  to  evaluate  the  effectiveness  before  the  system  has 
been  at  sea  for  many  years.  Since  this  manual  is  primarily  concerned  with  systems 
performance  effectiveness  in  its  entire  context,  the  following  discussion  considers 
the  analytic  models— -with  the  understanding  that  empirical  methods  will  always 
be  required  to  provide  Inputs  for  the  analysis. 

2.  Characteristics  of  an  Effectiveness  Model 

There  are  certain  general  characteristics  that  any  mathematical  model  should 
have  in  order  to  be  a  useful  tool  to  predict  effectiveness.  Despite  the  fact  that 
some  of  the  points  discussed  below  seem  almost  obvious  in  retrospect,  a  substantial 
number  of  existing  models  do  not  seem  to  possess  these  characteristics.  Hence, 
the  following  discussion  appears  warranted: 

(1 )  Independence  from  Design  Assumptions 

If  the  concept  of  effectiveness  is  to  be  applied  as  a  technical 
management  tool,  there  is  a  demand  that  the  effectiveness-analysis 
technique,  and  consequently  the  analytic  model,  be  capable  of 
evaluating  alternative  (or  modified)  system  designs  with  respect  to  a 
fixed  set  of  mission  models  and  variables.  To  whatever  extent  the 
analytic  model  presupposes  system  design  configurations  or  characteristics, 
the  model  is  not  able  to  evaluate  alternate  designs  that  are  not  within 
these  constraints,  and  hence  it  may  not  provide  a  basiB  foi  comparison 
or  optimization.  For  example,  if  the  analytic  model  is  built  in  terms 
of  a  given  system-design  configuration,  other  design  configurations 
may  be  inequitably  treated  if  subjected  to  the  same  analysis. 

(2 )  Usefulness  Throughout  the  System  Life  Cycle 

The  analytic  model  should  be  one  that  can  be  used  throughout  the  system 
life  cycle.  In  the  early  stages  of  the  cycle,  relatively  few  data 
are  available  on  the  statistical  or  performance  capability  of  the  system, 
and  a  substantial  number  of  assumptions  must  be  made  to  permit  the 
analysis.  As  the  life  cycle  progresses  through  design,  development, 
test,  and  implementation,  additional  design  and  sample  test  data  ordi¬ 
narily  become  available.  The  analytic  model,  therefore,  should  be 
designed  to  accommodate  these  changes  in  inputs,  and  yield  successive 
systems  performance  effectiveness  predictions  throughout  the  life  cycle, 
with  increased  confidence  in  the  results. 
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(3)  Realism  In  the  Analytic  Assumptions 

The  physical  and  mathematical  assumptions  upon  which  the  model  is 
founded  must  be  realistic  with  respect  to  the  expected  characteristics 
of  the  mission  and  system  operations.  There  is  a  great  temptation 
to  construct  analytic  models  based  more  on  mathematical  elegance  than  on 
realism. 

(4)  Tractabillty  of  the  Evaluation 

For  the  model  to  be  usable,  it  must  give  numerical  anewers  when 
exercised.  This  implies  the  following: 

•  The  model  must  be  quantitative  even  in  the  face  of  limited  data 

•  It  must  be  amenable  to  computation. 

Clearly,  this  model  characteristic  must  be  traded  off  against  the 
characteristic  of  realism.  The  art  of  modeling  consists,  in  large 
measure,  of  establishing  this  balance. 

3-  Selecting  an  Effectiveness  Model 

There  appear  to  be  three  fundamental  classes  of  considerations  that  enter  into 
the  selection  of  an  appropriate  effectiveness  model:  the  outputs  required  for 
system  management  and  optimization,  the  nature  of  the  systems  to  be  analyzed, 
and  the  mission  characteristics  to  be  employed. 

3.1  Output  Considerations 

The  definitions  of  the  variables,  requirements,  and  decision  criteria 
influence  the  selection  of  an  appropriate  effectiveness  model.  The  following 
questions  are  typical  of  those  which  must  be  answered: 

•  Can  the  system-oriented  performance  variables  be  identified  with  specific 
hardware,  or  are  they  more  closely  tied  to  overall  system  behavior, 
including  software? 

•  Was  an  iterative  methodology  employed  to  establish  the  requirements 
and  decision  criteria?  (The  requirements  on  the  model  themselves  could 
change  during  the  iterative  procedure,  and  these  changes  must  be 
incorporated. ) 

•  Is  there  one  principal  performance  variable  that  corresponds  to  one 
principal  system  function,  or  is  the  system  called  upon  to  do  many 
things? 

•  Will  the  utility  and  trade-off  data  permit  the  results  of  effectiveness 
analysis  to  be  expressed  in  terms  of  discrete  quantities,  or  will 
probability  distributions  be  required  to  describe  systems  performance 
effectiveness  adequately? 

•  Are  the  variables  binary  (success/failure)  or  multivalued? 
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3.2  System  Considerations 

System  considerations  concerning  the  choice  of  an  effectiveness  model  have 
their  greatest  effect  on  the  statistical  and  logical  assumptions  that  underlie 
the  model.  In  a  given  system,  it  may  be  uniquely  possible  to  identify  subsystems 
with  their  corresponding  functions,  and  in  such  a  case  the  effectiveness 
evaluation  is  simplified.  On  the  other  hand,  if  interaction  of  subsystem 
functions  is  expected,  particularly  with  degraded  modes  of  operation,  the 
model  must  incorporate  this  flexibility  of  interaction.  Additionally,  the  analytic 
model  often  incorporates  assumptions  concerning  the  statistical  behavior  of  the 
system.  These  assumptions  may  or  may  not  be  valid  for  the  system  in  question, 
and  they  may  or  may  not  be  consistent  with  the  available  data.  Finally,  the 
scope  and  complexity  of  the  system  must  be  considered.  The  delicate  balance 
between  tractability  and  realism  discussed  above  must  be  resolved  in  terms  of 
anticipated  system  size  (size  being  expressed  in  such  terms  as  the  number  of 
components). 

3-3  Mission  Considerations 

In  addition  to  mission  effects,  described  under  the  heading  Output 
Considerations  above,  a  series  of  representative  mission  profiles  also  must  be 
examined  as  part  of  the  model-selection  process. 

The  results  of  the  studies  discussed  elsewhere  In  this  report  are  closely  involved 
in  this  examination.  For  example,  is  the  system  operating  in  a  steady-state 
environment,  or  are  the  missions  short  compared  with  other  statistical  time 
parameters?  In  the  former  case  (e.g.,  a  long  cruising  mission),  an  equilibrium 
or  steady-stats  model  may  be  employed.  In  the  latter  case,  a  mission-sensitive 
model  is  required  in  whole  or  in  part. 

Again,  is  the  mission  function  carried  out  over  a  time  segment  (e.g.,  a 
hunter-killer  exercise),  or  is  a  point  mission  involved  (e.g.,  weapon  launch)? 

Are  there  one  or  several  critical  mission  segments?  Do  the  requirements  and 
decision  criteria  for  systems  performance  effectiveness  change  with  mission  mode? 

Do  the  reliability  and  maintainability  characteristics  of  the  subsystems  change 
as  a  function  of  a  mission  segment? 

The  answers  to  such  questions  will  help  3hape  tne  assumptions  that  are 
incorporated  in  the  analytic  model. 
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The  development  of  a  model  that  satisfies  the  conditions  cited  in  the  preceding 
section  is  at  best,  a  complicated  task.  However,  even  under  the  assumption  that 
such  a  model  is  realizable,  Its  range  of  application  without  some  types  of  modifi¬ 
cation  would  be  restricted.  This  is  due  to  the  nature  of  the  requirements,  diverse 
operating  conditions,  and  use  factors  that  generally  are  an  integral  part  of 
military  systems.  To  help  solve  this  problem,  an  approach  that  uses  a  system 
effectiveness  framework  has  been  developed.  In  this  approach,  the  basic  systems 
performance  effectiveness  elements  remain  constant  for  different  system  missions 
and  use  functions,  although  the  more  detailed  factors  underlying  the  basic  elements 
are  subject  to  change,  depending  on  the  particular  problem  analyzed. 
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Several  systems  performance  effectiveness  models  have  been  developed. 

Table  B-l  lists  equations  for  the  more  general  models  used  by  the  Armed  Forces. 

All  these  equations  concern  systems  performance  effectiveness,  but  each  approaches 
the  subject  In  a  different  manner,  reflecting  the  needs  of  the  individual  service. 

In  general,  the  Navy  translates  Its  terms  FAU  Into  the  analytic  terms  Pq 
and  PT;  the  terms  PU  and  A  are  similar,  respectively,  to  the  WSEIAC  terms  C  and 
AD.  The  following  general  equations  are  derived: 

ES  =  EA*  E 

and 

f(P,A,U)  =  f(Pc,PT)  *  f(A,D,C) 

It  should  be  noted  that  properly  constructed  Navy  and  WSEIAC  models  of  the 
sa:..c  system  carrying  out  the  same  mission  will  give  the  same  evaluation,  and  may 
even  be  mathematically  identical. 

Although  both  the  Navy  PAU  model  and  WSEIAC  model  are  usually  defined  as 
equations,  the  basic  Equations  B-l  and  B-5,  given  in  Table  B-l  can  only  be  used 
for  simple  systems  in  simple  missions,  even  if  the  equations  are  assumed  to  be 
in  matrix  form. 

However,  the  Navy  f(P^,Pq,)  model  is  more  adaptable  to  complex  systems  by 
virtue  of  its  computerized  treatment  of  the  variable  P,p. 

Several  systems  performance  effectiveness  models  are  reviewed  below.  Emphasis 
has  been  placed  on  describing  the  framework  of  each  model  rather  than  on  providing 
a  detailed  description.  The  referenced  documents  should  be  consulted  for  the  latter. 


4.1  The  Navy  Model 

The  first  term.  Performance  (P),  in  the  Navy  model  (PAU)*  can  be  expressed 
within  several  frames  of  reference.  In  the  single -mission  system,  the  expression 
is  derived  from  a  variety  of  measurements,  e.g. ,  area  destroyed,  tons  of  cargo 

l 

r*  Systems  Effectiveness,  compiled  by  Systems  Effectiveness  Branch,  Office 
of  Naval  Material,  January  1965.  (AD  659-520) 

Proceedings  of  the  NMSE  Systems  Performance  Effectiveness  Conference,  the 
NKSI  Systems  Performance  Effectiveness  Steering  Conti ttee,  April  1965. 

(AD  629-145) 

Proceedings  of  the  Second  NMSE  Systems  Performance  Effectiveness  Conference, 
the  NMSE  Systems  Performance  Effectiveness  Steering  Coanittee,  April  1^66. 

(AD  651-819) 


Proceedings  of  the  WC  Third  System  Performance  Effectiveness  Conference, 
the  AC  System  Performance  Effectiveness  Steering  Committee,  May  1967. 

(AD  660-422) 


L.  D.  Whitelock  and  P.  J.  Giordano,  The  Navy* a  Systems  Psrformance  Effectiveness 
Program,  Aeronautic  and  Space  Engineering  Manufacturing  Meeting,  Society  of 
Automotive  Engineers,  October  1966. 
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TAILE  1-1 

EQUATIONS  FOX  SYSTEMS  PERFORMANCE  EFFECTIVENESS  MOOELS 


Title 

Equation 

Explanation  of  Terms 

Tenn 

Explanation 

Havy  Systems  Performance 
Effectiveness 

% 

-  (PAU) 

(B-U 

Es 

Index  of  Systems  Performance  EffectlVeneso 

Es 

-  r<P.A,u) 

(B-2) 

p 

Index  of  System  Perfomance  -  a  numerical  Index  express'* 
ing  system  capability  assuring  a  hypothetical  100£ 
availability  and  utilization  of  performance  capability 

In  actual  operation 

A 

Index  of  System  Availability  -  a  numerical  Index  of 
extent  to  which  a  system  Is  ready  and  capable  of  fully 
performing  It*  assigned  mlsslon{3) 

U 

Index  of  System  Utilization  -  a  numerical  Index  of  the 
extent  to  which  the  performance  capability  of  the  system 
;ls  utilized  during  ti  e  mission 

Navy  Analytic  Systems 

Performance  Effectiveness 

ea 

-  (PCPT> 

ea 

Systems  Performance  Effectiveness 

ea 

-  f(PcPT) 

a 

Measure  of  Performance  Capability  *  a  Measure  of 
adequacy  of  design  and  system  degradation 

1 

Measure  of  Detailed  Tl.se  Dependency  -  a  measure  of 
availability  with  a  given  utilization. 

WSKIAC  System  Bffectlveneaa 

E 

(ADC) 

HjgSI 

E 

Quantitative  measure  of  systems  performance  >.rfectl venues 

E 

f(A,D,C) 

■ 

Measure  of  Availability  -  a  measure  of  the  condition 
of  a  system  at  the  start  of  a  mission  when  the  mission 

Is  called  for  at  unknown  (random)  point  In  time. 

1 

1 

Measure  of  Dependability  -  a  measure  of  the  system 
condition  during  thr  .  trfonnance  of  the  mission  given 
its  condition  (availability)  at  the  start  of  the 
mission. 

B 

1 

Measure  of  Capability  -  a  measure  of  the  results  of 
the  mission  Riven  the  condition  of  the  system  during 
the  mission  (dependability) 

Navy  at  Effectiveness 

Ec 

ES  f(P,A,U) 

ca  *  c0  c&  +  c0 

(B-7) 

Index  of  Coat  Effectiveness 

Cost  of  Acquisition  -  the  aggregated  costs  of  acquir¬ 
ing  the  system,  Including  prorated  development  costs. 

Coat  of  Ownership  -  the  aggregated  coats  of  operating 
and  maintaining  the  system. 

h.vy  i*.'!*nae  Effectiveness 

Ed 

V  «l  fP,A,U) 

*  ^  E=  ■ 

-- 

(B-8) 

Ed 

Index  of  Defense  Effectiveness 

V 

Index  of  Military  Worth 

Index  of  Time  Effectiveness 

..l  f  *  qulaltlon 

CA 

“  fl'af  Cs*'  af’  Cas' 

(B-9) 

°»t 

Acquisition  Tl*e  Costs  -  the  calendar  time  required 
to  acquire  the  system. 

c 

u 

Acquisition  Manpower  Costs  -  the  manpower  and  skill 
levelr  required  to  acquire  the  system 

Caf 

Acquisition  financial  Ousts  (Dollar)  -  the  dollar 
outlays  required  to  acquire  the  system,  Including 
the  dollar  costs  associated  with  manpower  and  Its 
acquisition  and  supporting  facilities. 

“4# 

Acquisition  Sipportlag.  Facilities  Cost*  -  the 
penalty  coat  to  other  systems  through  use  of  support 
facilities  by  the  system  during  acquisition  of  the 

system. 

.  t  f  .wnershlp 

C 

0 

f'  ,«•  cf 

(S-K'l 

Ownership  fenpewer  Cik.j  -  the  manpower  and  skill 
levels  required  to  operate  aid  maintain  the  system 

e-t 

Ownership  Financial  Costs  (Dollar)  -  the  dollar 
outlays  required  to  operate  and  maintain  the  uystre. 
Including  the  dollar  -oat#  neeo<* lated  with  aanpowe- 
and  Its  acquisition  *n4  supporting  faellttlea. 

1 _ 

i c- 

i 

_ 

ownership  Supporting  Forces  and  Facilities  '  ate  - 
tn*  penalty  teats  to  other  systems  through  require-  j 

men?  for  and  use  of  support  In*  facilities  an*!  forces  j 

in  '.ns  operation  and  »a intense*-*  cf  the  ayeter,  j 
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or  number  of  passengers  delivered,  emitters  located  and  identified.  Two  important 
conditions  apply:  (1)  the  measurement  standard  used  must  be  applicable  to  the 
parameter  used  to  determine  the  performance  level  and  (2)  the  answer  derived  from 
exercising  the  expression  must  be  used  with  caution  because,  with  other  than 
extremely  simple  systems,  the  achieved  performance  capability  1b  almost  always 
less  than  the  theoretical  performance  capability.  This  circumstance  occurs  because 
the  design-optimization  process  requires  that  some  trade  offs  be  made  to  achieve 
optimization  of  the  overall  system.  As  a  result,  even  for  the  relatively  simple, 
single -mission  system,  (P)  is  expressed  as  an  index  representing  the  ratio  of  the 
achieved  performance  level  to  the  theoretical  desired  level.  In  essence,  it  is 
a  figure  of  merit  even  under  the  assumption  of  absolute  availability  and  absolute 
utilization. 

In  the  case  of  the  multi-mission  system,  consideration  must  also  be  given  to 
the  interaction  of  two  Judgments.  The  first  comprises  the  assignment  of  •. sighting 
(importance)  factors  to  the  several  mission  modes  of  the  system  in  such  a  way 
that  their  sum  is  1.0.  The  second  comprises  the  determination  of  the  fraction 
of  the  system's  total  mission  time  that  will  be  devoted  to  each  of  the  several 
mission  modes. 

It  is  in  the  multi -mission  system  that  the  compelling  reason  for  using 
indexes  becomes  most  apparent.  Many  such  systems  have  completely  disparate 
standards  for  measuring  the  performance  of  their  various  mission  modes.  A  compari¬ 
son  or  aggregation  of  performance  indexes  that  use  different  measurement  standards 
can  not  be  attempted  validly.  For  example,  tonB  of  cargo  delivered,  area  destroyed, 
personnel  transported,  and  enemy  radar  sites  located  and  identified  cannot  logically 
be  compared  or  aggregated. 

The  second  term,  availability  (A),  is  more  complex  than  the  first.  Overall 
availability  is  relatively  easy  to  measure,  but  seperating  the  overall  value  into 
factors  of  reliability,  maintainability,  operability,  and  supportability  remains 
a  difficult  task — particularly  in  regard  to  pi*ediction  of  the  effect  of  uptime 
or  downtime.  Figure  2-1*  indicates  the  factors  that  contribute  to  availability 
according  to  the  Navy's  definition  of  the  term.  The  factors  associated  with  the 
man-module ( a )  in  the  system  are  not  now  quantifiable;  they  must  be  quantized 
(provided  numeric  representations  of  judgment  values)  and  indexes  or  figures  of 
merit  must  be  used  to  synthesize  availability. 

The  third  systems  performance  effectiveness  teru.,  Otllization  (U),  accents 
for  factors  that  are  introduced  by  the  tactical,  functional,  logistical,  and 
environmental  utilization  of  the  system;  all  four  are  a  function  of  the  operational 
doctrine  of  th*  system. 


*  Chapter  2,  page  6,  In  main  text. 
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Utilization  factors  represent  the  degradation  in  system  performance  caused 
by  mission  conditions.  The  following  are  some  examples: 

•  Loss  of  accuracy  of  an  optical  gun  sight  due  to  haze 

•  Increase  in  part  failure  rate  due  to  high  ambient  temperature 

•  Reduction  in  repair-part  availability  due  to  remoteness  of  location 
from  supply  depot 

•  Infrequent  use  of  search  radar  for  security  reasons 

The  utilization  factors,  except  for  analytic  exercise  of  the  model,  are 
relatively  constant.  However,  the  assigned'  values  will  change  whenever  operational 
goals  and  criteria  are  modified  in  the  process  of  achieving  consonance  between 
them  and  technical  goals  and  criteria. 

The  real  significance  of  the  utilization  factors  lies  in  their  ability  to 
be  varied  in  both  sensitivity  and  trade-off  analyses  for  optimizing  the  entire 
system  and  its  use.  The  systems  performance  effectiveness  model  thus  becomes 
a  tool  for  bringing  operational  and  technical  goals  and  criteria  into  agreement 
with  each  other. 

If  the  goals  and  criteria  are  not  in  agreement,  technical  managers  can  use 
the  models  to  demonstrate  to  their  operational  counterparts  the  desirability  of 
changing  the  operational  goals  and  criteria.  In  such  a  demonstration  the  utiliza¬ 
tion  factors  are  varied  to  show  the  impact  of  the  variances  on  the  index  of  the 
system's  performance  effectiveness.  If  this  exercise  does  not  demonstrate  the 
desirability  of  changing  the  o  'rational  goals  and  criteria,  the  technical  manager 
can  readily  understand  why  he  must  revise  his  goals  and  criteria  to  coincide  with 
those  of  the  operational  manager.  In  most  cases  it  will  become  clear  to  both  that 
revisions  are  necessary  on  both  sides  to  achieve  an  optimum  system. 

As  with  the  performance  and  availability  indexes,  the  variances  in  utilization 
indexes  must  be  evaluated  in  terms  of  cost  considerations  and  military-worth 
considerations.  Each  variance  of  a  factor  affects  the  other  factors  and  is  in 
turn  affected  by  variances  in  the  other  factors.  At  the  same  time,  each  variance 
of  a  factor  has  an  associated  cost  that  must  be  considered.  Only  when  all  factors 
have  been  considered  in  terms  of  mission  accomplishment  will  true  performance 
effectiveness  be  achieved  for  the  system. 

4.2  The  Air  Force  (WSEIAC)  Model 

This  section  summarizes  the  generalized  mission -oriented  system-effectiveness 
model  that  was  developed  by  Task  Oroup  II  of  WSEIAC*.  In  the  simple  case  in 
which  the  system  can  only  be  in  either  a  working  state  or  a  failed  state,  the 


em- 


Oommtttee  (WSEIAC),  Pinal  Report 
Effectiveness  mvision,  Air  Worse 


55  end  45856,  Jsnuary  1965. 
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measures  of  availability,  dependability,  end  capability  concern  the  following 
fundamental  questions: 

(1)  Is  the  system  working  at  the  start  of  the  mission? 

(2)  If  the  system  is  working  at  the  start  of  the  mission  will  it  continue 
to  work  throughout  the  mission? 

(3)  If  the  system  worked  throughout  the  mission,  will  it  achieve  mission 
success? 

Although  these  questions  represent  the  fundamental  approach  to  be  used  in 
evaluating  effectiveness  on  a  mission-oriented  basis,  they  are  too  simplified 
for  purposes  of  model  construction.  Moreover,  as  the  systems  considered  become 
more  complex--e.g. ,  there  are  more  than  two  possible  system  states — such  elements 
as  degraded  modes  of  operation,  multimission  requirements,  enemy  countermeasures, 
and  natural  environment  must  also  be  quantified  in  the  model. 

The  basic  effectiveness  model  can  be  divided  into  two  major  elements:  the 
probability  that  the  system  will  be  in  a  particular  state  at  mission-performance 
time,  and  the  effectiveness  of  the  system  when  it  is  in  that  state.  Thus  if 
effectiveness  is  quantified  by  a  probability  that  the  system  will  successfully 
meet  the  mission  objectives,  each  term  in  the  product  A  •  D  represents  the 
probability  that  the  system  will  be  in  a  particular  Btate,  and  the  corresponding 
term  in  the  C  vector  1b  the  effectiveness  of  the  system,  given  that  state. 

For  example, 

E  «  A  .  D  •  C 

■  ^ {Pfaystem  is  in  state  i]  •P[mission  objectives  are  met,  given  state  i ] } 
i-1 

For  some  types  of  systems  and  missions,  it  may  be  more  desirable  to  quantify 
effectiveness  by  some  performance  parameter  other  than  a  probability.  For  example, 
the  expected  miss  distance  for  a  missile  might  be  a  more  meaningful  performance 
parameter  than  the  probability  of  its  hitting  within  a  specified  area.  For  a 
reconnaissance  system,  the  average  amount  of  usable  information  might  be  appropriate. 
Figures  of  merit  for  these  forms  are  readily  usable  by  the  appropriate  quantifi¬ 
cation  of  the  C  vector. 

The  mission  model  proposed  by  the  WSEIAC  Task  Group  II  is,  in  essence,  more 
a  model  framework  for  effectiveness  evaluation  than  a  dlreotly  applicable  set  of 
equations.  This  generality  is  necessary  since  the  range  of  possible  systems, 
missions,  and  depth  of  analysis  precludes  the  speolfloatlon  of  any  single  model. 
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The  model  framework,  based  on  the  availability,  dependability,  and  capability 
factors,  allows  for  flexibility  in  application  by  an  appropriate  combination  of 
the  associated  elements.  In  Volume  3  of  the  Task  Group  II  report,  detailed 
examples  are  pi’esented  for  an  airborne  avionics  system,  an  intercontinental 
missile  system,  a  radar  surveillance  system,  and  a  spacecraft  system. 

The  level  of  detail  at  which  an  analysis  is  performed  will  depend  on  the 
information  and  data  available  and  on  the  purpose  of  the  evaluation.  Por  one 
study,  a  mean  repair  time  may  be  sufficient  input  for  the  availability  evaluation, 
while  for  another  study  such  factors  as  queuing  theory,  spare  parts  availability, 
maintenance  efficiency,  and  periodic -checkout  procedures  may  have  to  be  incorporated. 

There  are  still  many  different  areas  that  will  require  further  research. 

One  major  problem  is  to  develop  improved  techniques  to  convert  available  data 
into  the  appropriate  vector  and  matrix  elements  of  A,  D,  and  £.  Better  analytic 
and  computational  techniques  are  required  to  Incorporate  state  changes  and  those 
associated  capabilities  which  can  occur  over  a  continuous  Interval.  Such  factors 
as  state  occupancy  times  and  steady-state  behavior  may  be  Involved  in  such 
analyses.  Study  also  is  recommended  on  a  means  to  obtain  some  measure  of  "confi¬ 
dence"  in  the  results  of  the  effectiveness  evaluation,  both  in  the  probabilistic 
combination  of  estimates  and  in  guiding  the  decision  process  associated  with  the 
evaluation.  Computerized'  analytic  an&  simulation  methods  are  needed  for  complex 
systems  that  generate  a  very  large  number  of  system  states.  » ^ 

The  WSKtAC  model  framework,  or  similar  approach,  has  been  applied  to  several 
systems,  and  has  generally  been  found  to  be  a  reasonable  method  for  evaluating 
effectiveness  on  a  mission-oriented  basis.  Because  of  the  impetus  provided  by 
WSEIAC,  a  great  deal  of  research  is  being  sponsored  by  the  military  and  private 
agencies  in  order  to  improve  this  first  effort. 
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